# Maser Threshold Characterization by Resonator Q-Factor Tuning

Christoph W. Zollitsch<EMAIL_ADDRESS>Department of Chemistry,
Saarland University, Saarbrücken, 66123, Germany  Department of Physics &
Astronomy, University College London, Gower Street, WC1E 6BT, UK  London
Centre for Nanotechnology, University College London, 17-19 Gordon Street,
WC1H 0AH, UK  Stefan Ruloff  Department of Chemistry, Saarland University,
Saarbrücken, 66123, Germany  Yan Fett  Department of Chemistry, Saarland
University, Saarbrücken, 66123, Germany  Haakon T. A. Wiedemann  Department of
Chemistry, Saarland University, Saarbrücken, 66123, Germany  Rudolf Richter
Department of Chemistry, Saarland University, Saarbrücken, 66123, Germany
Jonathan D. Breeze  Department of Physics & Astronomy, University College
London, Gower Street, WC1E 6BT, UK  Christopher W. M. Kay
<EMAIL_ADDRESS>Department of Chemistry, Saarland University,
Saarbrücken, 66123, Germany  London Centre for Nanotechnology, University
College London, 17-19 Gordon Street, WC1H 0AH, UK

## Abstract

Whereas the laser is nowadays an ubiquitous technology, applications for its
microwave analogue, the maser, remain highly specialized, despite the
excellent low-noise microwave amplification properties. The widespread
application of masers is typically limited by the need of cryogenic
temperatures. The recent realization of a continuous-wave room-temperature
maser, using NV- centers in diamond, is a first step towards establishing the
maser as a potential platform for microwave research and development, yet its
design is far from optimal. Here, we design and construct an optimized setup
able to characterize the operating space of a maser using NV- centers. We
focus on the interplay of two key parameters for emission of microwave
photons: the quality factor of the microwave resonator and the degree of spin
level-inversion. We characterize the performance of the maser as a function of
these two parameters, identifying the parameter space of operation and
highlighting the requirements for maximal continuous microwave emission.

Figure 1: Schematic cross-section of the microwave cavity-resonator setup. a
Microwave cavity, holding the sapphire ring resonator. The schematic shows
half of the cavity to reveal the interior. The light blue sapphire ring is
held by two Teflon holders. On the top is the iris coupled waveguide port. The
coupling to the resonator is adjusted by a Teflon screw with a metal ring at
its tip. The $532\,\mathrm{n}\mathrm{m}$ optical pump laser is aligned along
the symmetry axis of the cylindrical resonator, while the static magnetic
field $B_{0}$ is perpendicular to it. Finally, the diamond sample is placed
inside the resonator. b Loaded $Q$-factor as a function of the resonator
coupling, showing the full range of coupling, with the diamond sample in the
resonator.

## Introduction

The first maser system was realized using ammonia molecules in the gas phase
Gordon1955  and applications for signal amplifiers, frequency standards or as
spectrometers were subsequently proposed. However, the need for cryogenic
and/or high vacuum environments restricted miniaturization and integration
towards more general applications. Maser research and development was mainly
focused on low-noise microwave receiving systems for deep-space antenna
networks Reid1973 , and several other maser systems based on ruby Makhov1958 ,
atomic hydrogen Goldenberg1960  or Rydberg atoms Moi1983  were realized. These
systems were all still subject to the same restrictions and the focus on
fundamental research into masers declined. The field was reinvigorated upon
the realization of a room-temperature pulsed maser in an optically pumped
crystal of pentacene-doped p-terphenyl, placed inside a high quality factor
microwave resonator Oxborrow_Nature2012 , Breeze_NComms2015 , Salvadori2017 .
Here, masing was achieved not only without the need of cryogenics or a high-
vacuum environment, but also with easily accessible optical pump rates.
Shortly afterwards, a proposal for a continuous-wave room-temperature maser in
optically spin polarized, negatively charged nitrogen vacancy (NV-) centers in
diamond Jin_NComms2015  and its consecutive experimental realization
Breeze_Nature2018  followed. The excellent low-noise amplification properties
of the maser have been demonstrated in recent work Sherman2022 ,
Koppenhofer2022 , Jiang2022_amp  and has found application in enhanced quantum
sensing of molecular spin ensembles Wu2022 . The search for other solid-state
maser materials, such as SiC Gottscholl2022  has continued, and an application
in quantum technology by creating a maser system based on Floquet states in Xe
atoms Jiang2022  has also been reported.

With the increased focus on room-temperature solid-state masers a quantitative
experimental characterization of the parameter space of operation supplements
the current endeavours to optimize the performance of such systems.
Understanding the behaviour of the maser performance, defined by the level-
inversion of the spin ensemble and the loaded $Q$-factor $Q_{\mathrm{L}}$ of
the resonator, can lead to higher output power maser systems. To guarantee
reproducibility and maximise the power output of maser-based technologies, a
complete understanding of the system parameter space, including the minimal
requirements to surpass the masing threshold, is required.

Here, we present an experimental setup that we exploited to investigate the
maser performance of a NV- spin ensemble hosted in diamond as a function of
the resonator quality factor and the degree of spin level-inversion. In the
resulting maser threshold diagram, we can clearly identify the threshold for
maser action, thereby obtaining a set of experimental boundary conditions for
the optimal operation of a maser system. Additionally, our optimized setup
yields the highest continuous-wave maser output power reported to date.

## Results and Discussion

### Experimental setup:

We use a cylindrical dielectric ring resonator made of sapphire to
deliver/detect resonant microwaves to/from the NV- centers contained in the
diamond host. A key parameter for the continuous maser emission is the loaded
$Q$-factor, $Q_{\mathrm{L}}$, of the resonator, which is defined by
$\nicefrac{{1}}{{Q_{\mathrm{L}}}}=\nicefrac{{1}}{{Q_{\mathrm{int}}}}+\nicefrac{{1}}{{Q_{\mathrm{ext}}}}$
with the internal $Q$-factor, $Q_{\mathrm{int}}$, and the external $Q$-factor,
$Q_{\mathrm{ext}}$. Although sapphire dielectric resonators exhibit low
dielectric losses Shtin2009 , radiative losses typically dominate and prevent
high internal quality factors. Hence, to suppress radiative losses the
resonator is placed inside a metal cavity. The cavity design is further
constrained by two conditions: (i) it has to fit between the poles of our
electromagnet system, which gives the static magnetic field $B_{0}$ used to
tune the energy levels of the NV- spins via the Zeeman interaction, and (ii)
the resonance frequency of the cavity containing the sapphire ring is required
to be within the $9-10\ \mathrm{G}\mathrm{Hz}$ (X-band) frequency range, to
allow a fast pre-characterization of the NV- spin ensemble by conventional
electron spin resonance (ESR). To this end, we designed a cylindrical cavity
made of oxygen-free high-thermal conductivity (OFHC) copper, plated with thin
layers of (first) silver and (second) gold to prevent oxidation of the metal
surfaces, thereby minimising additional resistive losses.

Figure 2: Schematic of the maser emission process. a - c The blue mirrors are
the optical analogue of a high-$Q$ microwave resonator, with one mirror being
fully reflective (dark blue) and one weakly transmissive (light blue); a
Energy states of the NV- spin ensemble aligned along $B_{0}$. The $\ket{-1}$
state is tuned below the $\ket{0}$ state by the Zeeman interaction, while the
optical pump generates a population inversion and spins spontaneously emit
microwaves. b Spontaneously emitted or thermally stimulated microwaves can
stimulate the emission of additional microwaves. c For sufficiently large
optical pump rates and a high $Q_{\mathrm{L}}$ coherent microwave emission is
established.

Figure 1 (a) shows a 3D schematic of the fully assembled cavity cut at a
symmetry plane to reveal the interior. The cavity has sample entries at
opposite ends of the cylinder symmetry axis for sample and optical access. In
the center of the cavity the resonator is held in place by two wire-frame
Teflon holders. The holders are machined to have minimal volume in order to
minimize additional dielectric losses. Microwave power is coupled in and out
via a single waveguide iris port on the top. This port represents the external
quality factor $Q_{\mathrm{ext}}$ and together with $Q_{\mathrm{int}}$ the
resonator coupling $k=\nicefrac{{Q_{\mathrm{int}}}}{{Q_{\mathrm{ext}}}}$ is
defined. $k$ is controlled via a Teflon screw with a metal ring at its tip.
Changing the coverage of the iris by the metal ring allows a continuous change
Accatino1994  from over-coupled to under-coupled: the regimes where
$Q_{\mathrm{L}}$ is dominated by external losses or intrinsic losses,
respectively. Figure 1 (b) shows the achievable $Q_{\mathrm{L}}$ as a function
of $k$ for our resonator-cavity system, loaded with the diamond sample. The
resonator parameters are extracted from microwave reflection measurements,
using a vector network analyzer. By fitting the microwave reflection as a
function of frequency with a Lorentzian model function the parameters for
resonance frequency, resonator coupling and internal quality factor are
determined. A description of the model function can be found in the methods
section. We use the TE01δ mode for our experiments, where the electric field
is mostly contained in the sapphire ring and the magnetic field is mostly
focused in the bore. Without a sample, the sapphire resonator has a resonance
frequency $\omega_{\mathrm{res}}/2\pi=9.25\,\mathrm{G}\mathrm{Hz}$ and the
$Q_{\mathrm{L}}=42,500$ when fully under-coupled. We define the resonator as
fully under-coupled when $k=0.0027$, where the Iris is no longer covered by
the metal tip of the Teflon screw. Here, $Q_{\mathrm{L}}$ remains unchanged
upon further extraction of the screw. For detailed cavity and resonator
dimensions see Supplementary Note 4 supplement .

### Maser working principles:

The process of continuous emission of microwave photons from the NV- centers
is schematically shown in Fig. 2 (a) to (c). The diamond hosting the NV-
centers is placed inside a high-$Q$ resonator, which is highly under-coupled
($k\ll 1$). The resonator may be pictured in analogue to its optical
counterpart the laser, with one perfectly reflective mirror and one weakly
transmitting mirror. The latter represents the iris coupled single microwave
port on the microwave cavity.

The applied static magnetic field $B_{0}$ lifts the degeneracy of the
$\ket{\pm\,1}$ states and, tunes the $\ket{-1}$ state energetically below the
$\ket{0}$ state such that the splitting $\hbar\omega_{\mathrm{M}}$ is resonant
with the microwave resonator frequency. The experiment is performed only on
one sub-set of NV- centers which are aligned with the external magnetic field
$B_{0}$. This provides the shown level structure, having the largest Zeeman
splitting of the energy levels and consequently the largest initial population
difference at Boltzmann equilibrium. By illuminating the NV- centers
continuously with a $532\,\mathrm{n}\mathrm{m}$ laser, the spin populations
which are initially at Boltzmann equilibrium are predominately pumped into the
$\ket{0}$ state Robledo_NJP2011 , resulting in a level-inversion (see Fig. 2
(a)). A description of the optical spin polarization process is found in
Supplementary Note 2 supplement . Finally, the laser polarization is required
to be aligned along the NV- defect axis to achieve most efficient pump rates
Kai-Mei2009 , Doherty2013 .

To trigger a collective stimulated emission, an initial photon with
$\hbar\omega_{\mathrm{M}}$ is required. This is provided either by an
externally applied seeding photon, due to spontaneous emission or thermal
photons (see Fig. 2 (b)). From this point an avalanche of stimulated photons
is created, forming a coherent microwave field inside the resonator (see Fig.
2 (c)). If the laser pump rate is sufficient to maintain the level inversion
and the resonator loaded quality factor is high enough to support a large
enough coherent microwave field, continuous microwave emission is achieved.

Figure 3: Maser emission spectrum. a Maser emission power
$\left|A_{\mathrm{M}}\right|$, in logarithmic units, as a function of
frequency and static magnetic field at $w_{\mathrm{L}}=430\,\mathrm{s}^{-1}$
and $Q_{\mathrm{L}}$ of about $33,500$. b $\left|A_{\mathrm{M}}\right|$ as a
function of frequency for two different magnetic field values, indicated by
the red and green dashed lines in a.

### NV defect axis alignment:

In our experimental setup, the diamond sample is held inside a quartz ESR
tube, supported between two additional quartz tubes fitted inside the first
one. The tube is inserted into the cavity and positioned such that the diamond
is located at the center of the sapphire ring where the magnetic component of
the microwave field is largest. The cavity with sample is mounted between the
poles of a electromagnet system. We connect either a conventional ESR
spectrometer, a vector network analyzer or a spectrum analyzer to the
microwave port of the cavity, to perform low-power microwave spectroscopy of
the NV- spin transitions or to study the maser emission. A goniometer is
attached to the sample tube, allowing a precise rotation of the quartz tube
containing the diamond with respect to the static magnetic field. Conventional
ESR as a function of $B_{0}$ and rotation angle is performed to find an
orientation of the diamond where the defect axis of a sub-set of NV- centers
is mostly parallel to the applied magnetic field. For such an orientation the
NV- spins feature an energy level scheme of the electronic ground state as
schematically shown in Fig. 2 (a) and the states $\ket{0}$ and $\ket{\pm 1}$
can be considered pure. This can be characterized by the frequency/magnetic
field splitting between the low-field
$\left(\ket{0}\,\mapsto\,\ket{+1}\right)$ and high-field
$\left(\ket{0}\,\mapsto\,\ket{-1}\right)$ transitions corresponding to twice
the zero-field splitting $D$. Away from this alignment the states become
mixed, resulting in a smaller splitting than $2D$ between the two allowed
transitions Mrozek2015 , Jeong2017  and a smaller maximal achievable spin
polarization. Without laser illumination, we find a maximal splitting of about
$205\,\mathrm{m}\mathrm{T}$ or $5.762\,\mathrm{G}\mathrm{Hz}$, agreeing well
with twice the zero-field splitting, $D$, of NV- centers in diamond.

### Maser emission spectrum characterization:

Having optimized the orientation of the NV- centers, we characterize the
performance of our maser setup by analyzing the microwave emission spectrum.
Figure 3 (a) shows the color encoded maser emission power
$\left|A_{\mathrm{M}}\right|$ as a function of frequency and static magnetic
field for a laser pump rate $w_{\mathrm{L}}=430\,\mathrm{s}^{-1}$ and with the
resonator fully under-coupled, $Q_{\mathrm{L}}\approx Q_{\mathrm{int}}$, which
gives a loaded quality factor $Q_{\mathrm{L}}$ of $33,500$. The three bright
lines represent the maser emission of the three 14N hyperfine transitions of
the NV- centers oriented along $B_{0}$. In this configuration, we achieve a
maximum maser emission power of $-56.5\,\mathrm{d}\mathrm{B}\mathrm{m}$. The
maser emission shows a finite frequency-magnetic field dispersion, where the
maser power is maximal in the center of the line. This is illustrated in Fig.
3 (b) which depicts the maser emission power as a function of frequency for
two fixed magnetic fields. The dispersion results from the hybridization of
the microwave resonator mode and the resonant NV- transition. The resonator
resonance frequency $\omega_{\mathrm{res}}/2\pi=9.12\,\mathrm{G}\mathrm{Hz}$
lies at the center of the middle maser emission line. The inverted spin
population causes a dispersive shift of the resonator frequency to lower
frequencies for the lower magnetic field emission line and respectively to a
shift to higher frequencies for the higher magnetic field emission line. The
maser dispersion can be described via a Tavis-Cummings model with an inverted
spin polarisation, where the frequency range covered by the maser emission
lines increases with increasing inversion.

Figure 4: Maser threshold diagram. a Maximal maser emission power
$\left|A_{\mathrm{M}}\right|$, in logarithmic units, of the central hyperfine
maser transition line as a function of laser pump rate $w_{\mathrm{L}}$ and
the loaded quality factor $Q_{\mathrm{L}}$ of the resonator. The white and
grey dashed lines represent the theoretical threshold for masing, with and
without temperature effects, respectively. b $\left|A_{\mathrm{M}}\right|$ as
a function of $Q_{\mathrm{L}}$ for two $w_{\mathrm{L}}$, indicated by the
purple and red dashed lines in a. c $\left|A_{\mathrm{M}}\right|$ as a
function of $w_{\mathrm{L}}$ at two $Q_{\mathrm{L}}$, indicated by the blue
and green dashed lines in a.

In order to determine the dependency of the maser output on the laser pump
rate $w_{\mathrm{L}}$ and the loaded quality factor $Q_{\mathrm{L}}$, both
were varied systematically. The latter is dependent on the resonator coupling
$k$ which is controlled by the iris screw. Figure 4 (a) shows the peak maser
power $\left|A_{\mathrm{M}}\right|$ of the central maser line as a function of
$w_{\mathrm{L}}$ (from $165\,\mathrm{s}^{-1}$ to $695\,\mathrm{s}^{-1}$) and
$Q_{\mathrm{L}}$ (from $14,000$ to $33,500$). Note, that after each change of
$w_{\mathrm{L}}$ we wait 45 min before starting measurements to allow the
diamond to reach a thermal equilibrium (see Supplementary Note 1 for details
supplement ). For low loaded quality factors, i.e. an over-coupled resonator,
there is no microwave emission for all laser pump rates studied and the signal
amplitude is represented by the noise floor of the spectrum analyzer. Note
that critical resonator coupling ($k=1$) is achieved at $Q_{\mathrm{L}}\approx
20,000$, marking the transition between over- and under-coupling Goeppl2008 .
With increasing $Q_{\mathrm{L}}$ a weak microwave emission is observed for
moderate to high laser pump rates. Here, the emission spectrum is broad and
governed by the amplification of thermal photons residing in the resonator
Jin_NComms2015 . In this region, the resonator losses are still too high to
allow the build-up of sufficient stimulated photons for continuous masing. For
$Q_{\mathrm{L}}>26,000$ and $w_{\mathrm{L}}>400\,\mathrm{s}^{-1}$ the rate of
stimulated emission exceeds the losses in the resonator and the spin system,
and continuous masing is established. Figure 4 (b) shows the evolution of the
peak maser power as a function of the loaded quality factor for two different
laser pump rates. Upon passing the threshold $Q_{\mathrm{L}}$ of $26,000$,
$\left|A_{\mathrm{M}}\right|$ increases until reaching an onset of saturation
for the highest $Q_{\mathrm{L}}$, where the final level of saturation depends
on the laser pump rate. A $Q_{\mathrm{L}}$ larger than $26,000$ further
promotes the stimulated microwave emission, thus reducing the threshold laser
pump rate required. For a fully under-coupled resonator with
$Q_{\mathrm{L}}=33,500$, continuous microwave emission is already achieved for
$w_{\mathrm{L}}>200\,\mathrm{s}^{-1}$. Figure 4 (c) shows the peak maser power
$\left|A_{\mathrm{M}}\right|$ as a function of the laser pump rate
$w_{\mathrm{L}}$ at two different loaded quality factors of the resonator. The
peak maser power increases with increasing laser pumping, reflecting the
achieved level-inversion. Consequently, for the largest $Q_{\mathrm{L}}$ and
highest $w_{\mathrm{L}}$, which guarantees a stable microwave field inside the
resonator and the largest level inversion, respectively, the highest maser
power of $-54.1\,\mathrm{d}\mathrm{B}\mathrm{m}$ is achieved. This is an
improvement of more than three orders of magnitude in maser power over the
initial report on a NV--based maser in Ref. Breeze_Nature2018,  with reported
$-90.3\,\mathrm{d}\mathrm{B}\mathrm{m}$. The threshold laser pump rate
$w_{\mathrm{th}}$ for masing can be derived from the steady-state Heisberg
equations of motion based on the coupled resonator-spin ensemble Tavis-
Cummings model, including dissipative channels for the resonator and spins,
and is given by Breeze_Nature2018 , Jin_NComms2015 , Kolobov1993

$w_{\mathrm{th}}=\frac{\eta}{T_{1}\left(\frac{g_{0}^{2}N}{\kappa_{0}\gamma}-1\right)}.$
(1)

Here, $T_{1}$ is the effective longitudinal relaxation time of the NV- spins,
$g_{0}$ gives the strength of the magnetic dipole coupling between a single
spin and a single microwave photon Haroche2013 , $N$ is the absolute number of
spins per hyperfine transition and per NV- defect axis and $\kappa_{0}$ and
$\gamma$ are the resonator and spin loss rates (HWHM), respectively. We
estimate $g_{0}/2\pi$ by finite element simulations of the magnetic field
profile inside the sapphire ring resonatorOskooi2010  to an average value of
$0.244\,\mathrm{Hz}$ and determine $\gamma/2\pi$ and $N$ via low-power
microwave spectroscopy of the low-field transition
$\left(\ket{0}\,\mapsto\,\ket{+1}\right)$ to $530.8\,\mathrm{k}\mathrm{Hz}\pm
7.7\,\mathrm{k}\mathrm{Hz}$ and $2.32\,\times\,10^{13}$, respectively. A
detailed derivation of these system parameters is found in Supplementary Note
3 supplement . The resonator loss rate is defined through $Q_{\mathrm{L}}$ as
$\kappa_{0}=\nicefrac{{\omega_{\mathrm{res}}}}{{2Q_{\mathrm{L}}}}$, which we
control by changing the resonator coupling. The scaling factor $\eta$ modifies
the optical pump rate defined for a two-level system within the framework of
the Tavis-Cummings model to take all seven energy levels involved in the
pumping of a single NV- spin into account. We extract $\eta=14.05$ from the
calculated spin-level inversion as a function of $w_{\mathrm{L}}$
Jin_NComms2015 , Breeze_Nature2018 , by solving the set of optical pump rate
equations in the steady-state Sherman2021 . For $T_{1}$ we explicitly take the
influence of an optical pump into account. In addition to polarizing spins,
the pump leads to excessive heating of the diamond, which decreases the
$T_{1}$ time Jarmola2012 . We determine $T_{1}$ for each $w_{\mathrm{L}}$,
finding $5.2\,\mathrm{m}\mathrm{s}$ for low $w_{\mathrm{L}}$ and a minimum of
$1.5\,\mathrm{m}\mathrm{s}$ for the highest $w_{\mathrm{L}}$, where a detailed
description of the $T_{1}$ dependence on $w_{\mathrm{L}}$ is given in
Supplementary Note 1 supplement . The white dashed line in Fig. 4 (a) gives
the masing threshold $w_{\mathrm{th}}$ as a function of the loaded quality
factor and is in excellent agreement with the experimentally found threshold
of our maser, when acknowledging the reduction of the relaxation time due to
high pump rates. For comparison, we include the expected threshold for a fixed
$T_{1}=5.2\,\mathrm{m}\mathrm{s}$ (grey dashed line), demonstrating the
significant influence of laser heating.

## Conclusion

To summarize, our characterization setup allows the resonator coupling to be
continuously and precisely adjusted between over- and under-coupled in
conjunction with the optical pump rate. This permits the detailed study of the
performance of the maser as a function of the rate of stimulated emission and
the degree of level-inversion. Control of these parameters enabled the first
experimental verification of the maser threshold equation over a wide
parameter space. Thus, the regions of microwave emission below the detection
limit, thermal photon amplification and continuous masing could be identified
in a NV- diamond maser.

Our results highlight an efficient operation of the maser is in the highly
under-coupled regime. Clearly, this limits the maser output power as only a
small fraction of the microwaves in the weakly coupled resonator can exit. For
applications, a higher output is essential and, hence, a larger coupling to
the resonator is required. The resulting threshold diagram suggests that
either higher pump rates or a resonator with a higher $Q_{\mathrm{int}}$ will
enable a larger coupling of the resonator. However, higher pump rates not only
require bulky laser systems but cause the sample to heat up. We demonstrate
that the $T_{1}$ relaxation time is shortened, thus reducing the spin
inversion. Indeed, it is clear for our results that pump rates can be reduced
to a level where small (e.g. $<1\,\mathrm{W}$) laser diodes can be employed
instead. Therefore, increasing the $Q$ factor of the resonator is a more
viable approach, although it is necessary to point out that this will decrease
the bandwidthSherman2022 . Finally, the optimized design yielded an increase
of the maximal maser output of more than three orders of magnitude compared to
the initial report on a NV--based maser from
$-90.3\,\mathrm{d}\mathrm{B}\mathrm{m}$ to
$-54.1\,\mathrm{d}\mathrm{B}\mathrm{m}$ Breeze_Nature2018 . We attribute this
mainly to an improved heat management of the diamond sample to limit a
reduction in $T_{1}$ time and consequently limit the increase of the masing
threshold. Thus it provides not only a blueprint for solid-state based maser
systems and but also sets a benchmark for future characterization and
optimization studies.

## Methods

Diamond Sample: Our diamond sample is of rhombic shape with its long axis
having about $5\,\mathrm{m}\mathrm{m}$, its short axis having about
$4\,\mathrm{m}\mathrm{m}$ and a thickness of $1\,\mathrm{m}\mathrm{m}$. The
diamond consists of natural abundance carbon and we estimate the total number
of NV- to $2.78\,\times\,10^{14}$ or $0.16\,$ppm (see Supplementary Note 3 for
details on the number of spins estimate supplement ). At room temperature and
no optical pump the NV- feature a
$T_{2}=25\,\mathrm{\SIUnitSymbolMicro}\mathrm{s}$, determined by pulsed ESR.

Laser Pumping: We use a $532\,\mathrm{n}\mathrm{m}$ Coherent Verdi V-5 laser
to optically pump the spin population of the NV- centers. The laser features a
spot size of about $4\,\mathrm{m}\mathrm{m}$ and hits the diamond sample on
its flat edge, an area of about $4\,\mathrm{m}\mathrm{m}\times
1\,\mathrm{m}\mathrm{m}$. We determine the laser pump rate as
$w_{\mathrm{L}}=\nicefrac{{\sigma
P_{\mathrm{pump}}}}{{A_{\mathrm{pump}}\hbar\omega_{\mathrm{pump}}}}$ (Ref.
Breeze_Nature2018, ), with the one-photon absorption cross-section for NV-
centers $\sigma=3.1\times 10^{-}{21}\,\mathrm{m}^{2}$, the pump laser power
$P_{\mathrm{pump}}$, the laser spot area $A_{\mathrm{pump}}$ and the laser
frequency $\omega_{\mathrm{pump}}$.

Microwave Spectroscopy: We study the microwave emission from the maser with a
Keysight N9020B MXA $10\,\mathrm{Hz}$ to $44\,\mathrm{G}\mathrm{Hz}$ spectrum
analyzer. To improve the SNR we pre-amplify the maser signal, using a Mini
Circuits Low Noise Amplifier ZX60-06183LN+ prior detection by the spectrum
analyzer. The low power microwave spectrocopy was carried out using an Anritsu
vector network analyzer MS46122B $1\,\mathrm{M}\mathrm{Hz}$ to
$43.5\,\mathrm{G}\mathrm{Hz}$. For conventional ESR measurements a Bruker
EMXplus spectrometer is used.

Resonator Coupling: We determine the microwave resonator coupling $k$ by
measuring the microwave reflection as a function of frequency with a VNA, far
detuned from the NV- centers. We fit the magnitude of the reflection
scattering parameter $\left|S_{11}\right|$ of the resonator for different
positions of the Iris screw, using a Lorentzian model function derived from an
equivalent circuit model, which depends on $k$ Wang2018 .

$\left|S_{11}\right|=A\left|\frac{k-1-iQ_{\mathrm{int}}\left(\frac{\omega}{\omega_{\mathrm{res}}}-\frac{\omega_{\mathrm{res}}}{\omega}\right)}{k+1+iQ_{\mathrm{int}}\left(\frac{\omega}{\omega_{\mathrm{res}}}-\frac{\omega_{\mathrm{res}}}{\omega}\right)}\right|.$
(2)

In addition, we can extract the internal quality factor $Q_{\mathrm{int}}$ and
via the relation for the resonator coupling
$k=Q_{\mathrm{int}}/Q_{\mathrm{ext}}$ the external Q-factor $Q_{\mathrm{ext}}$
and hence the loaded quality factor $Q_{\mathrm{L}}$, as well as the resonator
frequency $\omega_{\mathrm{res}}$ and its loss rate $\kappa_{0}$.

## Data Availability

The data that supports the findings of this study is available upon reasonable
request.

## Author Contribution

S.R., R.R. and C.W.M.K. designed and optimized the resonator. The experimental
setup and its automation was realized by S.R., Y.F., H.T.A.W., C.W.Z. and
C.W.M.K.. C.W.Z. performed the experiments and the data analysis with input
from J.D.B. and C.W.M.K.. C.W.Z. supported by C.W.M.K. wrote the manuscript
with input from all authors. C.W.Z, J.D.B. and C.W.M.K. conceived the study.

## Acknowledgment

This study is supported by EPSRC through EP/S000690/1 and EP/S000798/2 and the
Royal Society through URF$\backslash$R1$\backslash$191297\. We greatly
acknowledge the work and expertise of the workshop personnel Stefanie Porger
and Jens Wiegert at the University of Saarland for building the cavity and
components for it.

## Competing interests

The Authors declare no conflict of interests.

## References

## References

  * [1] Gordon, J. P., Zeiger, H. J. & Townes, C. H.  The maser—new type of microwave amplifier, frequency standard, and spectrometer.  _Physical Review_ 99, 1264–1274 (1955).
  * [2] Reid, M., Clauss, R., Bathker, D. & Stelzried, C.  Low-noise microwave receiving systems in a worldwide network of large antennas.  _Proceedings of the IEEE_ 61, 1330–1335 (1973).
  * [3] Makhov, G., Kikuchi, C., Lambe, J. & Terhune, R. W.  Maser action in ruby.  _Physical Review_ 109, 1399–1400 (1958).
  * [4] Goldenberg, H. M., Kleppner, D. & Ramsey, N. F.  Atomic hydrogen maser.  _Physical Review Letters_ 5, 361–362 (1960).
  * [5] Moi, L. _et al._ Rydberg-atom masers. i. a theoretical and experimental study of super-radiant systems in the millimeter-wave domain.  _Physical Review A_ 27, 2043–2064 (1983).
  * [6] Oxborrow, M., Breeze, J. D. & Alford, N. M.  Room-temperature solid-state maser.  _Nature_ 488, 353–356 (2012).
  * [7] Breeze, J. _et al._ Enhanced magnetic purcell effect in room-temperature masers.  _Nature Communications_ 6, 6215 (2015).
  * [8] Salvadori, E. _et al._ Nanosecond time-resolved characterization of a pentacene-based room-temperature maser.  _Scientific Reports_ 7, 41836 (2017).
  * [9] Jin, L. _et al._ Proposal for a room-temperature diamond maser.  _Nature Communications_ 6, 8251 (2015).
  * [10] Breeze, J. D., Salvadori, E., Sathian, J., Alford, N. M. & Kay, C. W. M.  Continuous-wave room-temperature diamond maser.  _Nature_ 555, 493–496 (2018).
  * [11] Sherman, A. _et al._ Diamond-based microwave quantum amplifier.  _Science Advances_ 8, eade6527 (2022).
  * [12] Koppenhöfer, M., Groszkowski, P., Lau, H.-K. & Clerk, A.  Dissipative superradiant spin amplifier for enhanced quantum sensing.  _PRX Quantum_ 3, 030330 (2022).
  * [13] Jiang, M. _et al._ Floquet spin amplification.  _Physical Review Letters_ 128, 233201 (2022).
  * [14] Wu, H. _et al._ Enhanced quantum sensing with room-temperature solid-state masers.  _Science Advances_ 8, eade1613 (2022).
  * [15] Gottscholl, A. _et al._ Superradiance of spin defects in silicon carbide for maser applications.  _Frontiers in Photonics_ 3 (2022).
  * [16] Jiang, M., Su, H., Wu, Z., Peng, X. & Budker, D.  Floquet maser.  _Science Advances_ 7, eabe0719 (2022).
  * [17] Shtin, N. A., Romero, J. M. L. & Prokhorov, E.  Theory of fundamental microwave absorption in sapphire ($\alpha$-al2o3).  _Journal of Applied Physics_ 106, 104115 (2009).
  * [18] Accatino, L. & Bertin, G.  Design of coupling irises between circular cavities by modal analysis.  _IEEE Transactions on Microwave Theory and Techniques_ 42, 1307–1313 (1994).
  * [19] For additional information see: Supplementary Material: Maser Threshold Characterization by Resonator Q-Factor Tuning.
  * [20] Robledo, L., Bernien, H., Sar, T. v. d. & Hanson, R.  Spin dynamics in the optical cycle of single nitrogen-vacancy centres in diamond.  _New Journal of Physics_ 13, 025013 (2011).
  * [21] Fu, K.-M. C. _et al._ Observation of the dynamic jahn-teller effect in the excited states of nitrogen-vacancy centers in diamond.  _Physical Review Letters_ 103, 256404 (2009).
  * [22] Doherty, M. W. _et al._ The nitrogen-vacancy colour centre in diamond.  _Physics Reports_ 528, 1–45 (2013).
  * [23] Mrózek, M. _et al._ Longitudinal spin relaxation in nitrogen-vacancy ensembles in diamond.  _EPJ Quantum Technology_ 2, 22 (2015).
  * [24] Jeong, K. _et al._ Understanding the magnetic resonance spectrum of nitrogen vacancy centers in an ensemble of randomly oriented nanodiamonds.  _Journal of Physical Chemistry C_ 121, 21057–21061 (2017).
  * [25] Göppl, M. _et al._ Coplanar waveguide resonators for circuit quantum electrodynamics.  _Journal of Applied Physics_ 104, – (2008).
  * [26] Kolobov, M. I., Davidovich, L., Giacobino, E. & Fabre, C.  Role of pumping statistics and dynamics of atomic polarization in quantum fluctuations of laser sources.  _Physical Review A_ 47, 1431–1446 (1993).
  * [27] Haroche, S. & Raimond, J.-M.  _Exploring the quantum. Atoms, cavities and
photons. Reprint of the 2006 hardback ed._ (Oxford Graduate Texts. Oxford:
Oxford University Press., 2013).

  * [28] Oskooi, A. F. _et al._ Meep: A flexible free-software package for electromagnetic simulations by the fdtd method.  _Computer Physics Communications_ 181, 687–702 (2010).
  * [29] Sherman, A., Buchbinder, L., Ding, S. & Blank, A.  Performance analysis of diamond-based masers.  _Journal of Applied Physics_ 129, 144503 (2021).
  * [30] Jarmola, A., Acosta, V. M., Jensen, K., Chemerisov, S. & Budker, D.  Temperature- and magnetic-field-dependent longitudinal spin relaxation in nitrogen-vacancy ensembles in diamond.  _Physical Review Letters_ 108, 197601 (2012).
  * [31] Wang, P. _et al._ Novel method to measure unloaded quality factor of resonant cavities at room temperature.  _Nuclear Science and Techniques_ 29, 50 (2018).

Supplementary Material: Maser Threshold Characterization by Resonator Q-Factor
Tuning

## Supplementary Note 1: Temperature Increase of the Diamond due to Optical
Pumping

The study of the maser output power as a function of optical pump rate
requires considerable laser powers. The highest pump rate
$w_{\mathrm{L}}=695\,\mathrm{s}^{-1}$ is achieved by a laser power of
$1.05\,\mathrm{W}$. Such high powers lead to excess heating of the diamond,
which cannot be dissipated fully in our measurement setup. As described in the
main text, the diamond sample is held inside a quartz ESR tube
($5\,\mathrm{m}\mathrm{m}$ outer diameter (OD)), supported between two
additional quartz tubes ($4\,\mathrm{m}\mathrm{m}$ OD) fitted tightly inside
the first one. Through the ESR tubes the diamond is indirectly thermally
connected to the sapphire ring as well as the tube holders on the cavity body,
which allows a finite thermal transport of the excess heating from the laser.
In addition, we introduce a weak nitrogen gas flow into the cavity body to
achieve an additional cooling effect. The nitrogen gas is connected to an
inlet on the waveguide and flows via the Iris into the cavity and out via gaps
in the ESR tube holders. Together, a finite thermal equilibrium in the diamond
will establish after some time. For the experiments presented in the main
text, we wait 45 minutes before a start of a measurement, after changing the
laser power. The longitudinal relaxation time $T_{1}$ exhibits a strong
temperature dependence Jarmola2012 . For temperatures above room-temperature
the relaxation rate $\nicefrac{{1}}{{T_{1}}}$ predominately increases with
$T^{5}$. As the maser threshold $w_{\mathrm{th}}$ is indirect proportional to
$T_{1}$ (see main text), the threshold will increase due to the effects of
laser heating.

Supplementary Figure 1: Temperature dependence of zero-field splitting and
relaxation time. a Color encoded microwave reflection $\left|S_{11}\right|$ as
a function of VNA frequency and static magnetic field, at a pump rate of
$695\,\mathrm{s}^{-1}$. b Effective loss rate $\kappa_{\mathrm{eff}}$ of the
resonator - spin ensemble hybrid system as a function of the static magnetic
field. The orange solid line is a fit of Eq. (S4) to the data.

To determine the temperature increase in our maser characterization setup, we
first determine the change of the zero-field splitting $\Delta D$ as a
function of optical pump rate, via low power microwave spectroscopy of the
low- and high-field NV- spin transitions, and is shown in Supplementary Figure
1 (a). With increasing pump rate the zero-field splitting is decreasing. We
define the room-temperature value of $D=2.88\,\mathrm{G}\mathrm{Hz}$ as the
reference for the relative shift $\Delta D$. Due to this temperature
dependence, NV centers are highly sensitive thermometers. The temperature
dependence can be described by a second order polynomial Ouyang2023,

$\Delta D=\Delta D_{\mathrm{cal}}+a_{1}T+a_{2}T^{2},$ (S1)

with coefficients
$a_{1}=3.279\,\times\,10^{-2}\,\mathrm{M}\mathrm{Hz}\mathrm{K}^{-1}$ and
$a_{2}=-1.787\,\times\,10^{-4}\,\mathrm{M}\mathrm{Hz}\mathrm{K}^{-2}$. With
the constant offset $\Delta D_{\mathrm{cal}}=5.9\,\mathrm{M}\mathrm{Hz}$ we
calibrate the model to produce a zero shift at room-temperature. With the
model function we can relate the measured $\Delta D$ to a temperature, as
shown in Supplementary Figure 1 (b).

Finally, we can find the corresponding $T_{1,\mathrm{eff}}$ times for each
optical pump rate used in the maser characterization. Supplementary Figure 1
(c) shows the calculated temperature dependence of $T_{1}$ (solid red line)
together with the $T_{1,\mathrm{eff}}$ for each pump rate used in the
experiment (symbols), ranging from $165\,\mathrm{s}^{-1}$ to
$695\,\mathrm{s}^{-1}$. We calculate the relaxation time following the
formalism presented in Jarmola2012,

$T_{1}=\left(A_{1}+\frac{A_{2}}{e^{\nicefrac{{\Delta}}{{k_{\mathrm{b}}T}}}-1}+A_{3}T^{5}\right)^{-1}.$
(S2)

The coefficients are $A_{1}=0.06\,\mathrm{s}^{-1}$,
$A_{2}=2.1\,\times\,10^{3}\,\mathrm{s}^{-1}$ and
$A_{3}=2.2\,\times\,10^{-11}\,\mathrm{s}^{-1}\mathrm{K}^{-5}$,
$\Delta=1.17\,\times\,10^{-20}\,\mathrm{J}$ is the dominant local vibrational
energy and $k_{\mathrm{b}}$ is the Boltzman constant.

## Supplementary Note 2: Optical Spin Polarization of the NV- Spin Ensemble

Supplementary Figure 2 shows the schematic electronic energy level scheme for
NV- spins at zero applied magnetic field Robledo_NJP2011 , Breeze_Nature2018 .
We use a green $532\,\mathrm{n}\mathrm{m}$ laser to excite the population from
the electronic ground state 3A2 into the excited state 3E. Hereby, all of the
spin triplet states $\ket{0}$ and $\ket{\pm 1}$ being pumped equally. The
excited triplet state decays via a red fluorescence back in the 3A2.
Additionally, the electrons from the excited $\ket{\pm 1}$ states undergo a
spin selective intersystem crossing via the singlet states 1A and 1E. From
there electrons decay into the 3A2 triplet ground state with about equal
rates. Due to the spin selective intersystem crossing in the excited state a
stable majority population of the $\ket{0}$ can be achieved, under continuous
laser pumping.

Supplementary Figure 2: Optical spin polarization scheme. Schematic of the
electronic energy levels of a NV- spin ensemble. The triplet states 3A2 and 3E
are considered to be quantized along a principal axis of the zero-field
splitting tensor. A continuous optical pump via a $532\,\mathrm{n}\mathrm{m}$
laser can achieve a majority population of the 3A2 $\ket{0}$ state, due to the
spin selective intersystem crossing in the excited 3E triplet.

## Supplementary Note 3: Microwave Spectroscopy of the NV- Spin Ensemble and
Number of Spins Estimation

In addition to the microwave emission spectrum of the maser, we investigated
the NV- spin transitions by low power microwave spectroscopy. In particular,
we studied the low-field electron spin resonance (ESR) transition
$\left(\ket{0}\,\mapsto\,\ket{+1}\right)$ as a function of microwave frequency
and magnetic field $B_{0}$ and recorded the microwave reflection $S_{11}$ with
a vector network analyzer (VNA). This transition is dominantly absorptive as
the optically spin polarized $\ket{0}$ is energetically lying below the
$\ket{+1}$ state, corresponding to a positive population difference.

Supplementary Figure 3 (a) shows the magnitude of the microwave reflection
$\left|S_{11}\right|$ as a function of VNA frequency and static magnetic
field, with an optical pump rate of $w_{\mathrm{L}}=695\,\mathrm{s}^{-1}$
applied to the spin ensemble. The horizontal purple line represents the
microwave resonator. At the center of the field region the resonator is
interrupted three times, where the resonator interacts with the NV${}^{-}1$
spin ensemble. The splitting in three lines is due to the hyperfine
interaction with the spin 1 N14 nuclei. On resonance the resonator experiences
a dispersive shift, indicating the onset of the strong coupling regime. We
extract the collective coupling strength $g_{\mathrm{eff}}$ between the
resonator and the spin ensemble to estimate the number of spins contributing
to the coupling. To this end, we fit the resonator microwave reflection for
each magnetic field point, using a Lorentzian model function derived from an
equivalent circuit model Wang2018 .

$\left|S_{11}\right|=A\left|\frac{k-1-iQ_{\mathrm{int}}\left(\frac{\omega}{\omega_{\mathrm{res}}}-\frac{\omega_{\mathrm{res}}}{\omega}\right)}{k+1+iQ_{\mathrm{int}}\left(\frac{\omega}{\omega_{\mathrm{res}}}-\frac{\omega_{\mathrm{res}}}{\omega}\right)}\right|.$
(S3)

Far detuned from the spin ensemble, we can extract the internal quality factor
$Q_{\mathrm{int}}$ and via the relation for the resonator coupling
$k=Q_{\mathrm{int}}/Q_{\mathrm{ext}}$ the external Q-factor $Q_{\mathrm{ext}}$
and hence the loaded quality factor $Q_{\mathrm{L}}$, as well as the resonator
frequency $\omega_{\mathrm{res}}$ and its loss rate $\kappa_{0}$ Wang2018 .
Over the full magnetic field range the loss rate changes, represented by an
effective loss rate $\kappa_{\mathrm{eff}}$ and shown in Supplementary Figure
3 (b). $\kappa_{\mathrm{eff}}$ shows three prominent peaks at the NV-
resonances, where the increased loss rate results from the hybridization of
the two systems, leading to an average between resonator and spin ensemble
loss rates $\kappa_{0}$ and $\gamma$, respectively. Using a coupled harmonic
resonator model, $\kappa_{\mathrm{eff}}$ can be described using Khan2021

Supplementary Figure 3: Microwave reflection of low-field NV- transition. a
Color encoded microwave reflection $\left|S_{11}\right|$ as a function of VNA
frequency and static magnetic field, at a pump rate of $695\,\mathrm{s}^{-1}$.
b Effective loss rate $\kappa_{\mathrm{eff}}$ of the resonator - spin ensemble
hybrid system as a function of the static magnetic field. The orange solid
line is a fit of Eq. (S4) to the data.

$\kappa_{\mathrm{eff}}=\kappa_{0}+\sum_{n=-1}^{+1}\frac{g_{\mathrm{eff},n}^{2}\gamma}{\Delta_{n}^{2}+\gamma^{2}}.$
(S4)

The sum is over all three hyperfine split ESR transitions with $n=-1,0,+1$ and
$\Delta_{n}=\frac{g_{\mathrm{NV}}\mu_{\mathrm{B}}}{\hbar}\left(B_{0}-B_{\mathrm{ESR},n}\right)$
gives the detuning from the individual ESR resonances. We use the same spin
loss rate for all transitions and find
$\gamma/2\pi=530.8\,\mathrm{k}\mathrm{Hz}\pm 7.7\,\mathrm{k}\mathrm{Hz}$. The
collective coupling strength is the strongest for the lowest field transition
and weakest for the highest field transition, which we attribute to some
degree of saturation of the spin ensemble while sweeping the magnetic field
over the transitions. The highest collective coupling achieved is
$357.8\,\mathrm{k}\mathrm{Hz}\pm 2.8\,\mathrm{k}\mathrm{Hz}$, pumping the NV-
with $695\,\mathrm{s}^{-1}$.

To amount for the microwave saturation we calculate the excitation rate due to
the microwave probe tone. The input microwave signal is heavily attenuated to
$-55.7\,\mathrm{d}\mathrm{B}\mathrm{m}$ at the input of the microwave cavity
to limit saturation of the spin transition. The microwave signal is further
attenuated by the insertion loss to the fully undercoupled resonator, given by
$IL=-10\log\frac{4k}{(1+k)^{2}}$ Jaynes1945, Moreno1948. Here, k is the
resonator coupling coefficient and is determined by fitting the resonator
reflection signal to $0.155\pm 0.002$ Wang2018 . This results in an insertion
loss of $3.33\,\mathrm{d}\mathrm{B}\mathrm{m}$, leading to a final microwave
power in the resonator of about $-59\,\mathrm{d}\mathrm{B}\mathrm{m}$. This
corresponds to an average number of photons $\left<n\right>$ of $2.87\times
10^{8}$ inside the resonator with a loaded quality factor of
$Q_{\mathrm{L}}=33,500$. The rate of excitation as well as for relaxation if
given in SI-units by Castner1959

$\begin{split}\left(\frac{dn}{dt}\right)_{\mathrm{MW}}&=\frac{1}{4}n\left(\frac{g_{\mathrm{NV}}\mu_{\mathrm{B}}}{\hbar}\right)^{2}B_{1}^{2}g\left(\omega-\omega_{\mathrm{ESR}}\right),\\\
\left(\frac{dn}{dt}\right)_{\mathrm{T1}}&=\frac{n-n_{0}}{T_{1}}.\end{split}$
(S5)

Here, $g_{\mathrm{NV}}$ gives the NV- g-factor, $\mu_{\mathrm{B}}$ the Bohr
magneton, $h$ the Planck constant, $T_{1}$ being the longitudinal relaxation
time, $n$ and $n_{0}$ are the relative spin population differences under
optical pumping and pure Boltzmann distribution without pumping, respectively,
both normalized by the total number of spins. The additional factor of
$\nicefrac{{1}}{{4}}$ at the microwave excitation rate takes into account that
the microwave energy can be absorbed by only one of the four NV- defect sub-
sets. $g\left(\omega-\omega_{\mathrm{ESR}}\right)$ gives the linewidth
function of the homogeneously broadened spin packets in the sample for which
we assume a Lorentzian lineshape Castner1959

$g\left(\omega-\omega_{\mathrm{res}}\right)=\frac{T_{2}}{\pi}\frac{1}{1+T_{2}^{2}\left(\omega-\omega_{\mathrm{ESR}}\right)^{2}},$
(S6)

where the spin-spin relaxation time is given by $T_{2}$ and
$\left(\omega-\omega_{\mathrm{res}}\right)$ gives the detuning from the
resonance frequency of the spin transition $\omega_{\mathrm{ESR}}$. We
estimate the rate of excitation via an averaged $B_{\mathrm{1,avg}}$ field
strength, determined by finite element simulations of the sapphire ring
resonator Oskooi2010  to $B_{\mathrm{1,avg}}=1.5\times 10^{-7}\,\mathrm{T}$.
On resonance and with $T_{2}=25\,\mathrm{\SIUnitSymbolMicro}\mathrm{s}$,
determined by conventional pulsed ESR, we estimate an excitation rate of
$93.0\,\mathrm{s}^{-1}$. At maximal laser pump rates,
$T_{1}=1.5\,\mathrm{m}\mathrm{s}$ (see previous section) and the rate of
relaxation results in $45.9\,\mathrm{s}^{-1}$. The population difference $n$
is calculated for an optical pump rate of $695\,\mathrm{s}^{-1}$, following
the rate equation formalism presented in Sherman2021, . With the rate of
excitation being about twice as high as the relaxation rate the spins are
partially saturated. Comparing the calculated spin populations with and
without a microwave drive we find that the microwave drive is reducing the
population difference of the $\ket{0}\,\mapsto\,\ket{+1}$ transition by
$19.1\,\%$. We scale the extracted value for the collective coupling
$g_{\mathrm{eff}}$ to remove the effects of saturation and yield
$g_{\mathrm{eff}}/2\pi=390\,\mathrm{k}\mathrm{Hz}\pm
3.1\,\mathrm{k}\mathrm{Hz}$.

From the finite element simulations of the resonator we are able to determine
an average single spin-photon coupling rate $g_{0}/2\pi=0.244\,\mathrm{Hz}$.
Assuming a homogeneous $B_{\mathrm{1}}$-field over the diamond sample, and
consequently a homogeneous distribution of $g_{0}$, the collective coupling is
given by $g_{\mathrm{eff}}=g_{0}\sqrt{S_{\mathrm{z}}}$, with $S_{\mathrm{z}}$
being the population difference between the two spin states. With the un-
saturated value for $g_{\mathrm{eff}}$ we find
$S_{\mathrm{z}}=2.57\,\times\,10^{12}$. To find the total number of spins for
a given hyperfine transition and single NV- defect axis, we scale
$S_{\mathrm{z}}$ up to $100\,\%$ spin polarization. At a pump rate of
$695\,\mathrm{s}^{-1}$ we calculate a normalized population difference of
$0.07$ between $\left(\ket{0}\,\mapsto\,\ket{+1}\right)$. Together, we find a
total number of spins per hyperfine transition and per NV- defect axis to
$N=3.67\,\times\,10^{13}$. To get a good agreement of the threshold equation
with the experimental data, we require to adjust the number of spins by a
factor of $0.63$, leading to a final number of spins of
$2.32\,\times\,10^{13}$. An accurate determination of the absolute number of
spins in a given sample is typically challenging and often only accurate to
the order of magnitude. Having estimated $N$ within a third to arrive at an
excellent agreement with the experimental data corroborates our method to
determine $N$.

## Supplementary Note 4: Microwave Cavity and Resonator Design Details

As described in the main text, the cavity’s purpose is to suppress the
radiation losses of the dielectric sapphire ring resonator. The dimensions are
chosen such that the quality factor of the resonator is as large as possible,
while still being able to be fitted inside of the poles of a normal conducting
magnet system. Supplementary Figure 4 shows the technical drawing of the
cavity, including all relevant dimensions. The body of the cavity is made of
OFHC copper and is gold platted to protect the copper surfaces from oxidation.
Prior gold platting a silver adhesion layer 1-2 $\mu$m thickness was put on
the copper followed by a 0.2-0.3 $\mu$m gold layer. Since the conductivity of
gold is lower than of pristine OFHC copper the thickness of the gold layer was
kept minimal. The cavity is coupled via an iris to a standard X-band
waveguide. The sapphire dielectric ring resonator is of cylindrical shape with
an outer diameter of $10\,\mathrm{m}\mathrm{m}$, an inner diameter of
$5.1\,\mathrm{m}\mathrm{m}$ and a height of $6\,\mathrm{m}\mathrm{m}$. We
determine the mode volume of the resonator to $V_{\mathrm{m}}=5.1\times
10^{-8}\mathrm{m}^{3}$, via finite element simulations Oskooi2010 . Together
with the volume of the diamond of about
$V_{\mathrm{dia}}=10^{-8}\mathrm{m}^{3}$ the filling factor of the resonator
with diamond sample is given by
$\eta=\nicefrac{{V_{\mathrm{dia}}}}{{V_{\mathrm{m}}}}=0.2$.

Supplementary Figure 4: Microwave Cavity Technical Drawing. The technical
drawing shows the dimensions of the gold platted copper cavity.

## References

  * [1] Gordon, J. P., Zeiger, H. J. & Townes, C. H.  The maser—new type of microwave amplifier, frequency standard, and spectrometer.  _Physical Review_ 99, 1264–1274 (1955).
  * [2] Reid, M., Clauss, R., Bathker, D. & Stelzried, C.  Low-noise microwave receiving systems in a worldwide network of large antennas.  _Proceedings of the IEEE_ 61, 1330–1335 (1973).
  * [3] Makhov, G., Kikuchi, C., Lambe, J. & Terhune, R. W.  Maser action in ruby.  _Physical Review_ 109, 1399–1400 (1958).
  * [4] Goldenberg, H. M., Kleppner, D. & Ramsey, N. F.  Atomic hydrogen maser.  _Physical Review Letters_ 5, 361–362 (1960).
  * [5] Moi, L. _et al._ Rydberg-atom masers. i. a theoretical and experimental study of super-radiant systems in the millimeter-wave domain.  _Physical Review A_ 27, 2043–2064 (1983).
  * [6] Oxborrow, M., Breeze, J. D. & Alford, N. M.  Room-temperature solid-state maser.  _Nature_ 488, 353–356 (2012).
  * [7] Breeze, J. _et al._ Enhanced magnetic purcell effect in room-temperature masers.  _Nature Communications_ 6, 6215 (2015).
  * [8] Salvadori, E. _et al._ Nanosecond time-resolved characterization of a pentacene-based room-temperature maser.  _Scientific Reports_ 7, 41836 (2017).
  * [9] Jin, L. _et al._ Proposal for a room-temperature diamond maser.  _Nature Communications_ 6, 8251 (2015).
  * [10] Breeze, J. D., Salvadori, E., Sathian, J., Alford, N. M. & Kay, C. W. M.  Continuous-wave room-temperature diamond maser.  _Nature_ 555, 493–496 (2018).
  * [11] Sherman, A. _et al._ Diamond-based microwave quantum amplifier.  _Science Advances_ 8, eade6527 (2022).
  * [12] Koppenhöfer, M., Groszkowski, P., Lau, H.-K. & Clerk, A.  Dissipative superradiant spin amplifier for enhanced quantum sensing.  _PRX Quantum_ 3, 030330 (2022).
  * [13] Jiang, M. _et al._ Floquet spin amplification.  _Physical Review Letters_ 128, 233201 (2022).
  * [14] Wu, H. _et al._ Enhanced quantum sensing with room-temperature solid-state masers.  _Science Advances_ 8, eade1613 (2022).
  * [15] Gottscholl, A. _et al._ Superradiance of spin defects in silicon carbide for maser applications.  _Frontiers in Photonics_ 3 (2022).
  * [16] Jiang, M., Su, H., Wu, Z., Peng, X. & Budker, D.  Floquet maser.  _Science Advances_ 7, eabe0719 (2022).
  * [17] Shtin, N. A., Romero, J. M. L. & Prokhorov, E.  Theory of fundamental microwave absorption in sapphire ($\alpha$-al2o3).  _Journal of Applied Physics_ 106, 104115 (2009).
  * [18] Accatino, L. & Bertin, G.  Design of coupling irises between circular cavities by modal analysis.  _IEEE Transactions on Microwave Theory and Techniques_ 42, 1307–1313 (1994).
  * [19] For additional information see: Supplementary Material: Maser Threshold Characterization by Resonator Q-Factor Tuning.
  * [20] Robledo, L., Bernien, H., Sar, T. v. d. & Hanson, R.  Spin dynamics in the optical cycle of single nitrogen-vacancy centres in diamond.  _New Journal of Physics_ 13, 025013 (2011).
  * [21] Fu, K.-M. C. _et al._ Observation of the dynamic jahn-teller effect in the excited states of nitrogen-vacancy centers in diamond.  _Physical Review Letters_ 103, 256404 (2009).
  * [22] Doherty, M. W. _et al._ The nitrogen-vacancy colour centre in diamond.  _Physics Reports_ 528, 1–45 (2013).
  * [23] Mrózek, M. _et al._ Longitudinal spin relaxation in nitrogen-vacancy ensembles in diamond.  _EPJ Quantum Technology_ 2, 22 (2015).
  * [24] Jeong, K. _et al._ Understanding the magnetic resonance spectrum of nitrogen vacancy centers in an ensemble of randomly oriented nanodiamonds.  _Journal of Physical Chemistry C_ 121, 21057–21061 (2017).
  * [25] Göppl, M. _et al._ Coplanar waveguide resonators for circuit quantum electrodynamics.  _Journal of Applied Physics_ 104, – (2008).
  * [26] Kolobov, M. I., Davidovich, L., Giacobino, E. & Fabre, C.  Role of pumping statistics and dynamics of atomic polarization in quantum fluctuations of laser sources.  _Physical Review A_ 47, 1431–1446 (1993).
  * [27] Haroche, S. & Raimond, J.-M.  _Exploring the quantum. Atoms, cavities and
photons. Reprint of the 2006 hardback ed._ (Oxford Graduate Texts. Oxford:
Oxford University Press., 2013).

  * [28] Oskooi, A. F. _et al._ Meep: A flexible free-software package for electromagnetic simulations by the fdtd method.  _Computer Physics Communications_ 181, 687–702 (2010).
  * [29] Sherman, A., Buchbinder, L., Ding, S. & Blank, A.  Performance analysis of diamond-based masers.  _Journal of Applied Physics_ 129, 144503 (2021).
  * [30] Jarmola, A., Acosta, V. M., Jensen, K., Chemerisov, S. & Budker, D.  Temperature- and magnetic-field-dependent longitudinal spin relaxation in nitrogen-vacancy ensembles in diamond.  _Physical Review Letters_ 108, 197601 (2012).
  * [31] Wang, P. _et al._ Novel method to measure unloaded quality factor of resonant cavities at room temperature.  _Nuclear Science and Techniques_ 29, 50 (2018).


# Object Memory Transformer for Object Goal Navigation

Rui Fukushima1, Kei Ota2,3, Asako Kanezaki2, Yoko Sasaki1 and Yusuke
Yoshiyasu1 1Rui Fukushima, Yoko Sasaki and Yusuke Yoshiyasu are with National
Institute of Advanced Industrial Science and Technology (AIST), Japan
rui.fukushima, y-sasaki<EMAIL_ADDRESS>Ota and Asako
Kanezaki are with Tokyo Institute of Technology, Tokyo, Japan
<EMAIL_ADDRESS>Ota is with Information Technology R&D Center,
Mitsubishi Electric Corporation, Kanagawa, Japan.
<EMAIL_ADDRESS>
###### Abstract

This paper presents a reinforcement learning method for object goal navigation
(ObjNav) where an agent navigates in 3D indoor environments to reach a target
object based on long-term observations of objects and scenes. To this end, we
propose Object Memory Transformer (OMT) that consists of two key ideas: 1)
Object-Scene Memory (OSM) that enables to store long-term scenes and object
semantics, and 2) Transformer that attends to salient objects in the sequence
of previously observed scenes and objects stored in OSM. This mechanism allows
the agent to efficiently navigate in the indoor environment without prior
knowledge about the environments, such as topological maps or 3D meshes. To
the best of our knowledge, this is the first work that uses a long-term memory
of object semantics in a goal-oriented navigation task. Experimental results
conducted on the AI2-THOR dataset show that OMT outperforms previous
approaches in navigating in unknown environments. In particular, we show that
utilizing the long-term object semantics information improves the efficiency
of navigation.

##  I Introduction

Object goal navigation (ObjNav) is a task of navigating an agent toward a
target object given by a word with reference to the first-person view [1, 2].
To solve the task, a navigation agent is required to learn highly nonlinear
mappings from images and words to actions. Unlike in game-playing tasks such
as GO, where Deep Reinforcement Learning (DRL) agents defeated human
professionals [3, 4], DRL agents’ performances on ObjNav are still far behind
from that of average humans [2].

The main challenge in ObjNav is that the task requires complex scene
understanding and can sometimes become long-horizon. In many cases, the target
object is not visible from the start position and thus the agent needs to get
close to the object by understanding the surrounding environment to plan its
paths from the first-person views in which the goal may not be present.
Further, these views drastically change according to the agent’s moves and
sometimes do not contain any meaningful information. For example, when the
agent faces to a white wall, there is no visual clue in the view to decide the
next move. As a result, the learning agent may easily be stuck and pivot at
the same place almost forever or may take an unexpectedly none-smooth
trajectory that repeatedly arrives at the same locations back and forth.
Another difficult point in ObjNav is its diversity: the trained agent model is
required to generalize to unknown environments with different room structures,
scenes, objects, etc., which makes this task even harder.

Recent DRL navigation approaches for ObjNav have achieved remarkable
improvements in performance using semantic and spatial knowledge about
objects, such as object relationship, spatial object context, and object class
hierarchy [1, 2, 5, 6, 7, 8]. However, the knowledge used in these studies is
independent from the temporal context, which means that it lacks the
information about the objects that are previously seen within the episode or
it does not respect the order in which objects appeared. Although some of
these studies have used Recurrent Neural Networks (RNNs), such as LSTM, it is
not clear if they can understand the ordering of long-range sequences [9].

Our goal is to investigate the effects of long-term observation of scenes and
objects in ObjNav, and how we can effectively use them to improve navigation
performance. To this end, we propose Object Memory Transformer (OMT) that
consists of the following two key ideas, detailed in Fig. LABEL:figurelabel.
Firstly, we encode the current scene and object to fixed dimensional feature
vectors, and store them in an external memory named Object-Scene Memory (OSM).
This memory is capable of storing scene observations and object semantics
throughout a long-term sequence in a temporally consistent manner. Then, a
Transformer network attends to the crucial scenes and objects, which are
stored in OSM, and produces useful features for the RL agent. We test our
model on publicly available synthetic indoor environments, and show that OMT
is able to navigate by taking efficient routes toward the target object. We
also investigate whether our memory architecture improves the performance and
efficiency in the ObjNav task.

##  II Related Work

###  II-A Object-goal Navigation

Yang et al. [1] proposed the first navigation RL agent to tackle ObjNav, using
Graph Convolution Networks (GCNs) [10] for incorporating scene priors. Since
then, various studies have attempted to use semantic knowledge about objects
to improve generalization performance to unseen targets and environments. In
particular, the approaches that combine semantics and spatial knowledge about
objects have achieved superior performance [2, 5, 6, 7, 8]. However, in ObjNav
a long-term sequence of observations has been rarely considered or exploited.
In this paper, we propose a new external memory called Object-Scene Memory
(OSM) to store the long-term time series of observed objects, and then use
Transformer [11] to extract useful features for ObjNav.

###  II-B Memory Based Network and Transformer

Recurrent Neural Networks (RNNs) [12] have been commonly used to capture time-
series information. However, RNNs are known to be insufficient to handle long-
term time-dependency. Therefore, a mechanism to store variables in an external
memory outside the network has attracted attentions [13, 14, 15, 16]. In the
field of natural language processing (NLP), a method called Transformer [11]
has been proposed and has shown its ability to process long sentences with
many words based on the self-attention mechanism. The transformer architecture
has become the de facto standard in NLP and has recently been applied to
computer vision problems [17, 18].

###  II-C Navigation agents with memory

Savinov et al. [19] proposed a memory architecture called Topological Memory,
referring to landmark-based navigation in animals. Khan et al. [20] used the
Differentiable Neural Computer (DNC) [16] in grid-world navigation. Zhu et al.
[15] extended Neural Episodic Control (NEC) and proposed Episodic
Reinforcement Learning with Associative Memory (ERLAM) [21] to improve sample
efficiency. ERLAM associates related experience trajectories by considering
the relationship between states. Fang et al. [22] proposed Scene Memory
Transformer (SMT) that stores long-term scene observations for none goal-
directed navigation tasks. Du et al. [23] proposed a Tentative Policy that
avoids deadlock and loop by utilizing attention mechanisms for attending to
all past actions stored in memory. Unlike these works, our Object-Scene Memory
stores long-term object semantics that can support the agent to take an
efficient trajectory to reach the target object in ObjNav.

Figure 2: Architecture of OMT that consists of four steps: 1) Feature
Extractor extracts visual features $\bar{\bm{v}}_{t}$ from RGB image
$\bm{v}_{t}$ using ResNet-50. It also encodes the relationship between target
object ${w}_{g}$ and the objects observed in the scene,
$W=\\{{w}_{1},...,{w}_{N^{\mathrm{obj}}}\\}$. This is done by computing the
word embeddings of those objects using word2vec and cosine similarities
between them, which will be assigned to the values in the object context grid,
$\bm{o}_{t}$. 2) OSM stores scene appearance $\bar{\bm{v}}_{t}$ and object
semantics $\bm{o}_{t}$ for the last $T$-history length, and fuses both of
those feature vectors into $\bm{m}_{t}$ using three learnable networks
$f_{v},f_{o},$ and $f_{m}$. We apply this operation to all data stored in OSM,
and obtains a set of features $M_{t}=\\{\bm{m}_{t-T-1},...,\bm{m}_{t}\\}$; 3)
Transformer then takes $M_{t}^{\mathrm{TE}}$ so that the model can learn
useful temporal feature representations for ObjNav, which are conditioned by
the target object; 4) Finally, an A3C-based controller outputs action
$\bm{a}_{t}$ to move in the 3D indoor environment.

##  III Proposed Method

In this section, we describe our approach to learning a navigation agent for
the ObjNav task. We start with explaining the task settings, and then
introduce our novel model architecture that consists of two key components: 1)
Object Scene Memory (OSM) that stores a history of observed scenes and
objects; 2) Object Memory Transformer (OMT) that takes the history of
observations from OSM, and produces useful features that will be taken as
input to the subsequent RL agent.

###  III-A Task Setting

ObjNav is a task that considers a setting where an agent is placed at a random
initial location in a 3D indoor environment, and required to reach a target
object defined in the target class set $G=\\{g_{0},...,g_{M}\\}$ while
minimizing the number of steps. Each task $\tau\in\mathcal{T}$ is represented
by $\tau=\\{e_{i},p_{i},g_{i}\\}$, where an environment $e_{i}$ is given from
a set of rooms $E=\\{e_{0},\dots,e_{N}\\}$, a starting position $p_{i}$, and a
target object $g_{i}\in G$. At each state, the agent takes a RGB image in a
first-person view, which we call scene, and objects in the form of bounding
boxes, which exist in the scene. We use the terms scene and object in this
manner.

The action space from which the agent takes an action at each time step
consists of the following nine commands: $\mathcal{A}=\\{$Move Forward, Move
Backward, Move Right, Move Left, Rotate Right, Rotate Left, Look Up, Look
Down, Done$\\}$. The agent moves on the discretized scene space divided by
$0.5\,\mathrm{m}$, and the rotation and the vertical tilt angle are $45$
degrees and $30$ degrees, respectively.

Each episode is considered to be success if the agent chooses Done action at
positions where the target object is visible, otherwise it is considered to be
failure. Here, visible becomes true when the target object is within threshold
distance, which we set to be $1.5\,\mathrm{m}$, from the agent.

###  III-B Model Architecture

The proposed model architecture is shown in Fig. 2. The agent model is based
on Transformer for handling the long-term history of object semantics, which
is inspired by Druon et al. [2] and Fang et al. [22]. Our method consists of
1) Feature Extractor, 2) Object-Scene Memory, 3) Transformer, and 4)
Controller. We describe the above components in details.

####  III-B1 Feature Extractor

The feature extractor takes the current scene $\bm{v}_{t}$ and object words
$W=\\{{w}_{1},...,{w}_{N^{\mathrm{obj}}}\\}$ and ${w}_{g}$ that specify the
objects shown in the current scene and the target object, and outputs the
features of the current scene and the target object. To extract the visual
feature, we use pre-trained ResNet-50 [24] that takes the $400\times 300$ RGB
image $\bm{v}_{t}$ and produces $2048$-dimensional visual feature
$\bar{\bm{v}}_{t}\in\mathbb{R}^{2048}$. The semantic feature of the target,
which is specified by the word ${w}_{g}$, is converted into $300$-dimensional
word embedding vector $\bar{\bm{w}}_{g}\in\mathbb{R}^{300}$ using word2vec
[25]. We do not update the parameters of these feature extractors during the
whole process.

In order to efficiently extract objects’ semantics in the current view, we use
the object context grid representation (object grid) proposed in [2]. The
object grid $\bm{o}_{t}$ consists of an array with $16\times 16$ cells,
$\bm{o}_{t}\in\mathbb{R}^{16\times 16}$. We assign a value to each cell that
indicates the similarities between the target object and the detected objects.
The positions to which these values are assigned correspond to the centers of
the bounding boxes of the detected objects. The similarities between the
target object and the detected objects are calculated by the cosine similarity
as:

$\bm{o}_{i,j}=\frac{\bar{\bm{w}}_{i,j}\cdot\bar{\bm{w}}_{g}}{||\bar{\bm{w}}_{i,j}||||\bar{\bm{w}}_{g}||},$

where $\bar{\bm{w}}_{g}$ and $\bar{\bm{w}}_{i,j}$ denote the word embedding
vectors of the target and an object whose center of bounding box detected at a
location corresponding to $i$-th row and $j$-th column of the object grid
$\bm{o}_{t}$, respectively. When no object located at $\bm{o}_{i,j}$, the
value becomes to be $0$. As for the object set $W$, we assume the ground truth
object labels and bounding boxes for all the objects in the view are available
following [2, 5].

####  III-B2 Object-Scene Memory

Next, we introduce Object-Scene Memory (OSM) that has two functionalities: 1)
storage of the visual appearance of the scene $\bar{\bm{v}}_{t}$ and the
object semantics $\bm{o}_{t}$; 2) extraction of useful features from the
stored data. Focusing on the first one, OSM can be considered as a ring-buffer
that can store $T$-history length. We prepare this ring-buffer for scene
appearance $\bar{\bm{v}}_{t}$ and the object semantics $\bm{o}_{t}$ (see Fig.
2). Secondly, OSM fuses the spatial and object semantics to the
300-dimensional fused vector $\bm{m}_{t}\in\mathbb{R}^{300}$. To that end, OSM
employs three feature extractors $f_{v},f_{o},f_{m}$ that take scene
appearance $\bm{v}$ and object semantics $\bm{o}$ as input. They fuse $\bm{v}$
and $\bm{o}$ to produce fused features $\bm{m}$ as:

$\bm{m}_{t}=f_{m}(f_{v}(\bar{\bm{v}}_{t}),f_{o}(\bm{o}_{t})).$ (1)

We do this operation for all time frames of OSM, producing a set of fused
features, $M_{t}=\\{\bm{m}_{t-T-1},...,\bm{m}_{t}\\}$.

####  III-B3 Transformer

Transformer receives features stored in memory and produces useful features to
the subsequent RL agent. We implemented temporal encoding and provided it to
Transformer so that it can get a reference to temporal ordering of the memory.

Temporal Encoding.  Since we hypothesize that temporal information can improve
the performance in the ObjNav task, we make use of the order of what the agent
has observed in an episode by utilizing the temporal encoding. Among different
implementations are proposed, we specifically use the trigonometric functions
proposed in [11]. Let $i$ be the current dimension of temporal encoding. Also,
let $T$ be the size of $M$. Then, the memory with temporal order encoded,
$M^{TE}$, can be formulated as:

$\displaystyle{\mathit{M}_{(\mathit{pos},2i)}^{TE}}$ $\displaystyle=$
$\displaystyle\bm{m}^{pos}_{t(2i)}+\sin\left(\frac{pos}{10000^{2i/T}}\right)$
(2) $\displaystyle{\mathit{M}_{(\mathit{pos},2i+1)}^{TE}}$ $\displaystyle=$
$\displaystyle\bm{m}^{pos}_{t(2i+1)}+\cos\left(\frac{pos}{10000^{2i/T}}\right),$
(3)

where $pos\in{\\{0,...,T-1\\}}$ and $\bm{m}^{pos}_{t}$ represent the location
of the memory slot and feature $\bm{m}_{t}$ located at the $pos$-th slot of
memory, respectively.

Encoder-Decoder networks. The temporally encoded scene and object context
features are then fed into the Transformer encoder-decoder network [11] to
obtain a useful representation for the RL agent. Transformer employs a
technique called self-attention to determine the relevance of each element to
other elements in a temporal data and outputs contextualized features.

The output of OSM along with temporal encoding $M^{\mathit{TE}}$ is taken as
an input to the Tranformer encoder $f_{\mathrm{enc}}$, and the encoder
extracts spatiotemporal and object semantics in the memory. Then, the decoder
$f_{\mathrm{dec}}$ produces useful representations $\bm{x}$ acquired from the
attention between the encoded memory and the target object to the controller.

$\displaystyle\bm{x}$ $\displaystyle=$ $\displaystyle
f_{\mathrm{dec}}(\bm{w}_{g},f_{\mathrm{enc}}(M^{TE}))$ (4)

Note that the encoder takes masked attention for both self-attention and
source-target attention when the memory slot is empty, i.e., when the current
time step is shorter than the size of the OSM.

####  III-B4 Controller

Finally, the features computed by the Transformer are passed to the
controller. We specifically use Asynchronous Advantage Actor-Critic (A3C) [26]
for an RL agent. In our implementation, the model has two fully-connected
layers and outputs both Q-values and the $|\mathcal{A}|$-length vector that
has probability of each categorical action.

Overall, the policy at time $t$ can be expressed as:

$\displaystyle s_{t}$ $\displaystyle=$
$\displaystyle(\bm{v}_{t},\bm{w_{g}},M_{t-1})$ (5) $\displaystyle a_{t}$
$\displaystyle\sim$ $\displaystyle\pi(a_{t}\mid s_{t},\bm{\theta}),$ (6)

where $s_{t}$ and $a_{t}$ are the state and action at time $t$ and
$\bm{\theta}$ are the model parameters.

###  III-C Reward Design

The RL agent is trained to maximize the expected return defined as the
expected cumulative reward $\mathbb{E_{\pi}}[\sum_{t=0}^{T}\gamma^{t}r_{t}]$.
We define the reward function in Eq. (7), which is a mixture of the one
proposed in [27] and [28] (details are described in [2]).

$r=\begin{cases}5.0&\text{if success}\\\ S_{bbox}&\text{if $S_{bbox}$ is the
highest in the episode}\\\ -0.01&\text{otherwise},\end{cases}$ (7)

where $S_{bbox}$ is computed by the ratio of the bounding box area in the
field of view, and $S_{bbox}=1$ when the entire field of view is covered by
the target object111If $S_{bbox}$ is not used, the agent immediately selects
the “$Done$” signal to avoid a decrease in the cumulative reward due to the
huge state space and reward sparsity, resulting in a failure of learning. To
avoid this, pre-training by imitation learning [6, 7, 23] or meta learning [8,
29] could be used..

The first condition encourages the agent to accomplish the task, and the
second condition encourages the agent to move closer to the target object. The
third condition promotes to learn to navigate in the shortest path as the
agent receives a penalty at each time step and tries to minimize it during
training.

##  IV Experimental Settings

This section briefly describes the experimental settings including the
dataset, evaluation metrics, and baseline models we used in the experiments to
validate the proposed method.

###  IV-A Dataset

We use the AI2-THOR framework (The House Of inteRactions) [30], which consists
of $120$ different photo-realistic 3D indoor environments for our experiments.
We use $80$ environments for training, i.e., $20$ environments from each of
$4$ categories: Kitchen, Living room, Bedroom, Bathroom. The target class $G$
for each category is manually selected in advance from predefined $146$
objects class as follows: Kitchen: {Toaster, Microwave, Fridge, Coffee
Machine, Garbage Can, Bowl}; Living room: {Pillow, Laptop, Television, Garbage
Can, Bowl}; Bedroom: {House Plant, Lamp, Book, Alarm Clock}; Bathroom: {Sink,
Toilet Paper, Soap Bottle, Light Switch}. Objects are placed in the room at
random locations determined by the AI2-THOR framework.

After training, we test the model in novel environments which are not included
in the training environments to evaluate its generalization capability.
Specifically, we used $20$ different environments, i.e., $5$ rooms from each
of $4$ categories. Note that when the AI2-THOR environments were created, some
targets were either not exist in the environment or not visible from any
possible locations. In such cases, they were omitted from the target set for
that environment.

###  IV-B Evaluation Metrics

We employ success rate (SR) and success weighted by path length (SPL) [31] for
our evaluation metrics. SR is defined as
$\frac{1}{N}\sum_{i=1}^{N}\mathbb{I}_{\rm goal}^{i}$, and SPL is defined as
$\frac{1}{N}\sum_{i=1}^{N}\mathbb{I}_{\rm goal}^{i}\frac{d^{\ast}}{d_{i}}$,
where $N$ represents the total number of evaluation episodes and
$\mathbb{I}_{\rm goal}^{i}$ is a success indicator, which is $1$ if the $i$-th
episode is successfully finished, and $0$ otherwise. The success condition is
described in Sec. III-A. The symbols $d^{\ast}$ and $d_{i}$ stand for the
shortest path length in a given scene from the initial position to the target
position, and the path length of $i$-th episode, respectively. In short, we
can say a policy is good if it achieves high score on both SR and SPL.

###  IV-C Models

We used the following models to compare the performance.

####  IV-C1 Random

This model was implemented so that it can be shown how complex the
experimental environments setup is. The agent uniformly samples one action
from the set of actions $\mathcal{A}$ at each step.

####  IV-C2 Scene Prior (SP)

This is an implementation of Scene Prior (SP) [1], which is the simplest
method of implementing semantic information of objects, implemented by [2]. SP
uses Graph Convolution Networks (GCNs) to extract object semantics, where each
node represents object class and edge represents relationship such as “next”
or “under/on.” Our model differs from SP in dealing with observations: SP does
not use spatiotemporal context of objects while ours does using OSM and OMT.
SP takes four staked frames of scenes as input, while GCNs take only the
current observation.

####  IV-C3 Baseline

We use the model proposed in [2] for the baseline, as this method also employs
the object context grid calculated with the ground truth object detector.
While this method also uses semantic and spatial knowledge about objects as
input, it does not explicitly utilize temporal information. Therefore, we can
evaluate the effectiveness of our memory structure by comparing the results
against this method. Note that this model takes four frames of observations as
input. We also test against the model with three LSTM layers, which enables
the model to handle temporal information. The LSTM layers take a single frame
of observation as input, and the object grid is fused with it.

####  IV-C4 OMT

We evaluate three models for our proposed method, whose size of the memory are
$\\{4,16,32\\}$, respectively, to see the relationship between history length
and its performance. We report the result with one layer of Transformer, but
we have also obtained similar results with two layers. Please refer to Sec.
III for more details.

##  V Experimental Results

In this section, we try to quantify the performance of OMT. Specifically, we
try to answer the following questions:

  * •

How does OMT perform on ObjNav in terms of SR and SPL compared against
previous approaches?

  * •

Which components of OMT contribute to the performance gain?

###  V-A Performance of OMT

Comparison with baselines  First, we evaluated the performance of the proposed
model. TABLE I shows the comparison between different approaches using the two
different metrics. We can clearly see that our method outperforms prior works
in both SR and SPL metrics. The result of Random shows that the ObjNav task
considers a huge navigation space and the agent cannot easily navigate to the
goal state by chance.

Comparing the two Baseline models, Baseline showed SR and SPL of around $60\%$
and $20\%$, respectively. In addition, Baseline-LSTM $3$ layers, which can
handle time-series information, showed a slight performance gain, i.e.,
$1.70\%$ higher SR and $3.13\%$ higher SPL than Baseline.

Focusing on our model, it achieves the best performance in terms of SR and
SPL. Our model with $4$ histories, which has the same history length as
Baseline, shows higher SR ($71.13\%$) than Baseline and Baseline-LSTM $3$
layers. Furthermore, when the history length is long, our agent model is more
likely to take the shortest path. In fact, our model with $32$ histories shows
the best SPL ($27.51\%$), which suggests that long-term histories can improve
navigation efficiency by choosing a shorter path to the target.

TABLE I: Evaluation results on unknown scenes and known objects

Method |  SR [$\%$]. |  SPL [$\%$]
---|---|---
Random | 6.10 | 1.30
SP [1] | 18.24 | 4.39
Baseline [2] | 61.55 | 20.83
Baseline - LSTM 3 layer [2] | 63.25 | 23.96
OMT - 32 hist. |  69.39 (0.16) |  27.51 (0.10)
OMT - 16 hist. |  70.00 (1.56) |  27.12 (0.22)
OMT - 4 hist. |  71.13 (1.42) |  26.66 (0.10)

Comparison with SOTA We also compared the performance against the current
state-of-the-art (SOTA) techniques [23, 5] based on the SR and SPL metrics
reported in the papers. Note that the experimental settings used in those
works are not exactly the same as ours. Thus some care must be taken when
comparing each other.

Qui et al. [5] proposes a navigation method that leverages a hierarchical
object relationship in the ObjNav task. While the method is tested on the
different grid step size ($0.25\,\mathrm{m}$) from ours ($0.5\,\mathrm{m}$),
its action space is $|\mathcal{A}|=5$. Thus it explores smaller action space
and makes the task easier. The reported scores are SR of $65.3\%$ and SPL of
$21.1\%$. Hence, we can conclude that our method works better than [5], as our
method achieves better performance for both of the two metrics on a harder
problem (action space is bigger).

Du et al. [23] proposes the memory-augmented tentative policy networks to
promote the agent to escape from deadlock states, such as looping or being
stuck. Although we could not compare SPL as their definition is different from
ours, the reported SR of $69.3\%$ is $2\%$ lower than our best result. Thus,
we can conclude that our OMT outperforms this SOTA method as well.

TABLE II: Results without each component.

Method | 32 hist. | 4 hist
---|---|---
SR [$\%$] |  SPL [$\%$] |  SR [$\%$] |  SPL [$\%$]
OMT(full) | 69.39 | 27.51 | 71.13 | 26.66
w/o Object Memory | 67.99 | 25.49 | 71.23 | 26.69
w/o Scene Memory | 65.38 | 23.15 | 62.60 | 25.10
w/o Scene | 62.30 | 24.28 | 65.59 | 24.81
w/o Temporal Encoding | 59.69 | 21.95 | 61.41 | 23.29
w/o Transformer | 64.27 | 25.43 | 68.84 | 25.71

###  V-B Ablation study—Effects of each component

To analyze what component of our method contributes to the performance gain,
we conducted an ablation study by removing one component from the full model
and measuring the performance in the two metrics as in Sec. V-A. The results
are shown in TABLE II. The model w/o Object Memory removes the context grid,
which encodes object semantics in the current scene, from OSM and directly
passes to the Controller. Likewise, for a model w/o Scene Memory, the current
scene is also concatenated. On the other hand, the w/o Scene, the current
scene is not used, and only the context grid is used as input. In the model of
w/o Temporal encoding, the temporal order is given by a soft manner like [22]
with using an exponential function which is concatenated with visual features.
In the w/o Transformer model, Transformer is replaced with two fully connected
layers and ReLU function.

To evaluate how much scene information (first-person views) contributes to the
navigation performance, we first see the results of w/o Scene Memory and w/o
Scene. To our surprise, we can see that both methods perform surprisingly well
even without scene information, whose scores are comparable with the baseline
model with LSTM. This shows that a decent level of navigation is possible only
using spatiotemporal knowledge of objects without recognizing a whole scene
image. Next, we removed the object memory from the full model and saw SR
dropped $1.40\%$ for $32$ histories, but it was $0.1\%$ higher than $4$
histories. On the other hand, when all the components are incorporated into
our model with $32$ histories, it shows the highest SPL score. This indicates
that navigation efficiency is the consequence of interacts between scene,
object, and memory. Especially, the temporal encoding of Transformer was found
to have the most prominent impact on navigation performance, indicating that
the temporal ordering of histories in memory is vital for ObjNav.

###  V-C Qualitative study

To qualitatively show the effectiveness of our key idea in using the long-term
history in ObjNav, we plotted agents’ trajectories for the episodes where the
start and goal are far from each other. In these cases, it is likely that
there are obstacles in its route and that the target is not visible from the
start location.

Fig. 3 shows the top-down maps with the trajectories of Baseline, Baseline-
LSTM $3$ Layer, OMT-$4$ hist., and OMT-$32$ hist. Focusing on the Fridge
results (left column), Baseline and OMT-$4$ histories move straight toward or
move left to get close to the target from the start, but are unable to move
due to an obstacle. In contrast, OMT-$32$ histories once got stuck in an
obstacle, but is able to get out of it and reach the target. We believe this
is because OMT can perceive the continuity and changes of observations over
time. In particular, the OMT recognizes its deadlock state when the same
observations continue to be perceived over a long time. Similarly in
Television (middle column), OMT-$32$ histories can recover to move to goal
position even it gets stuck in bumping into the desk before the television,
while Baseline and Baseline-LSTM $3$ layer select Done signal from outside the
threshold range. In Pillow (right column), where the agent is unable to see
the target from the start location because of the wall in the middle, OMT-$32$
histories once gets stuck in a loop at the beginning of the episode, but is
able to get out of it and reached the target while Baseline stuck in a
rotation loop until the episode exceeds the maximum number of steps. These
results suggest that our proposal of utilizing the long-term information can
help guide the agent in complex ObjNav tasks, where the agent cannot see the
goal object at the start position, especially in settings where the distance
between the start and goal is long.

Figure 3: Qualitative results. The top-down map shows the trajectories of
different methods. The initial location of the agent is shown as a black dot,
and the target object is circled by a yellow box. Trajectories are indicated
by purple squares of varying density. The thicker color, the more number of
times the agent stays at the location. The number above each top-down map
shows the total number of steps in this episode.

##  VI Conclusion

In this paper, we investigated how long-term histories of objects and the
first-person views can improve navigation performance in ObjNav task. To this
end, we presented Object Memory Transformer (OMT) that utilizes the histories
of information that an agent observes during navigation. The proposed method
consists of two key components: 1) Object-Scene Memory (OSM) that enables to
store long-term scene and object semantics, and 2) Transformer that attends to
salient objects in the sequence of previously observed scenes and objects. Our
experimental evaluation demonstrated that OMT can achieve state-of-the-art
performance on the AI2-THOR benchmark compared against prior works that do not
use long-term histories or use them in different manner. Additionally, we
showed that just using the long-term histories does not help improve the
performance, but combining them with temporal encoding is essential.

## Acknowledgment

This paper is in part based on the results obtained from a project, JPNP20006,
commissioned by the New Energy and Industrial Technology Development
Organization (NEDO).

## References

  * [1] W. Yang, X. Wang, A. Farhadi, A. Gupta, and R. Mottaghi, “Visual semantic navigation using scene priors,” _arXiv_ , vol. abs/1810.06543, 2018.
  * [2] R. Druon, Y. Yoshiyasu, A. Kanezaki, and A. Watt, “Visual object search by learning spatial context,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 1279–1286, 2020.
  * [3] D. Silver, A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, S. Dieleman, D. Grewe, J. Nham, N. Kalchbrenner, I. Sutskever, T. Lillicrap, M. Leach, K. Kavukcuoglu, T. Graepel, and D. Hassabis, “Mastering the game of go with deep neural networks and tree search,” _Nature_ , vol. 529, pp. 484–503, 2016.
  * [4] A. P. Badia, B. Piot, S. Kapturowski, P. Sprechmann, A. Vitvitskyi, Z. D. Guo, and C. Blundell, “Agent57: Outperforming the Atari human benchmark,” in _ICML_ , ser. PMLR, H. D. III and A. Singh, Eds., vol. 119. PMLR, 13–18 Jul 2020, pp. 507–517.
  * [5] Y. Qiu, A. Pal, and H. Christensen, “Learning hierarchical relationships for object-goal navigation,” _ArXiv_ , vol. abs/2003.06749, 2020.
  * [6] H. Du, X. Yu, and L. Zheng, “VTNet: Visual transformer network for object goal navigation,” in _ICLR_ , 2021.
  * [7] Y. Lv, N. Xie, Y. Shi, Z. Wang, and H. T. Shen, “Improving target-driven visual navigation with attention on 3d spatial relationships,” _arXiv_ , vol. abs/2005.02153, 2020.
  * [8] B. Mayo, T. Hazan, and A. Tal, “Visual navigation with spatial attention,” in _CVPR_ , 2021.
  * [9] R. Pascanu, T. Mikolov, and Y. Bengio, “On the difficulty of training recurrent neural networks,” _ICML_ , no. PART 3, pp. 2347–2355, 2013.
  * [10] T. N. Kipf and M. Welling, “Semi-supervised classification with graph convolutional networks,” _arXiv_ , vol. abs/1609.02907, 2016.
  * [11] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Nips_ , pp. 5999–6009, 2017.
  * [12] R. Jozefowicz, W. Zaremba, and I. Sutskever, “An empirical exploration of Recurrent Network architectures,” _ICML_ , vol. 3, pp. 2332–2340, 2015\.
  * [13] J. Oh, V. Chockalingam, S. Singh, and H. Lee, “Control of memory, active perception, and action in minecraft,” _ICML_ , vol. 6, pp. 4067–4089, 2016\.
  * [14] A. Graves, G. Wayne, and I. Danihelka, “Neural turing machines,” _arXiv_ , vol. abs/1410.5401, 2014.
  * [15] A. Pritzel, B. Uria, S. Srinivasan, A. P. Badia, O. Vinyals, D. Hassabis, D. Wierstra, and C. Blundell, “Neural episodic control,” _ICML_ , vol. 6, pp. 4320–4331, 2017.
  * [16] A. Graves, G. Wayne, M. Reynolds, T. Harley, I. Danihelka, A. Grabska, S. G. Colmenarejo, E. Grefenstette, T. Ramalho, J. Agapiou, A. P. Badia, K. M. Hermann, Y. Zwols, G. Ostrovski, A. Cain, H. King, C. Summerfield, P. Blunsom, K. Kavukcuoglu, and D. Hassabis, “Hybrid computing using a neural network with dynamic external memory,” _Nature_ , vol. 538, no. 7626, pp. 471–476, 2016.
  * [17] S. Khan, M. Naseer, M. Hayat, S. W. Zamir, F. S. Khan, and M. Shah, “Transformers in vision: A survey,” 2021.
  * [18] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _ECCV_ , 2020, pp. 213–229.
  * [19] N. Savinov, A. Dosovitskiy, and V. Koltun, “Semi-parametric topological memory for navigation,” _arXiv_ , pp. 1–16, 2018.
  * [20] A. Khan, C. Zhang, N. Atanasov, K. Karydis, V. Kumar, and D. D. Lee, “Memory augmented control networks,” in _ICLR_ , 2018.
  * [21] G. Zhu*, Z. Lin*, G. Yang, and C. Zhang, “Episodic reinforcement learning with associative memory,” in _ICLR_ , 2020.
  * [22] K. Fang, A. Toshev, L. Fei-Fei, and S. Savarese, “Scene memory transformer for embodied agents in long-horizon tasks,” _CVPR_ , vol. 2019-June, pp. 538–547, 2019.
  * [23] H. Du, X. Yu, and L. Zheng, “Learning Object Relation Graph and Tentative Policy for Visual Navigation,” _ECCV_ , pp. 19–34, 2020.
  * [24] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” _CVPR_ , pp. 770–778, 2016.
  * [25] T. Mikolov, K. Chen, G. Corrado, and J. Dean, “Efficient estimation of word representations in vector space,” _ICLR Workshop_ , pp. 1–12, 2013.
  * [26] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu, “Asynchronous methods for deep reinforcement learning,” in _Proceedings of The 33rd International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, M. F. Balcan and K. Q. Weinberger, Eds., vol. 48. New York, New York, USA: PMLR, 20–22 Jun 2016, pp. 1928–1937.
  * [27] Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei, and A. Farhadi, “Target-driven visual navigation in indoor scenes using deep reinforcement learning,” in _ICRA_ , jul 2017, pp. 3357–3364.
  * [28] X. Ye, Z. Lin, H. Li, S. Zheng, and Y. Yang, “Active Object Perceiver: Recognition-Guided Policy Learning for Object Searching on Mobile Robots,” _IROS_ , pp. 6857–6863, 2018.
  * [29] M. Wortsman, K. Ehsani, M. Rastegari, A. Farhadi, and R. Mottaghi, “Learning to learn how to learn: Self-adaptive visual navigation using meta-learning,” _2019 IEEE/CVF CVPR_ , pp. 6743–6752, 2019.
  * [30] E. Kolve, R. Mottaghi, W. Han, E. VanderBilt, L. Weihs, A. Herrasti, D. Gordon, Y. Zhu, A. Gupta, and A. Farhadi, “AI2-THOR: An Interactive 3D Environment for Visual AI,” _arXiv_ , 2017.
  * [31] P. Anderson, A. X. Chang, D. S. Chaplot, A. Dosovitskiy, S. Gupta, V. Koltun, J. Kosecka, J. Malik, R. Mottaghi, M. Savva, and A. Zamir, “On evaluation of embodied navigation agents,” _ArXiv_ , vol. abs/1807.06757, 2018.

## Appendix

###  VI-A Implementation Details

Our framework is implemented by using PyTorch, as well as Transformer was
implemented according to provided PyTorch package. RMSprop was used for the
optimization. The learning rate was $7\times 10^{-4}$, linearly decreasing to
$0$ in the last epoch. We use one layer in the encoder and decoder, and five
multi heads attention in Transformer. The model was trained with $80$ threads
with $25$ million frames in total with four NVIDIA V100 GPUs. For the
evaluation the agent is randomly located at the beginning of each episode and
expected to reach the target object within $300$ steps. we terminate the
episode if the total number of steps exceeds it, and consider it to be
failure. We evaluate our model with $250$ episodes on each target from target
class sets $G$.