# Engineering quantum states from a spatially structured quantum eraser

Carlo Schiano  Dipartimento di Fisica, Università di Napoli Federico II,
Complesso Universitario di Monte S. Angelo, Via Cintia, 80126 Napoli, Italy
Bereneice Sephton  Dipartimento di Fisica, Università di Napoli Federico II,
Complesso Universitario di Monte S. Angelo, Via Cintia, 80126 Napoli, Italy
Roberto Aiello  Dipartimento di Fisica, Università di Napoli Federico II,
Complesso Universitario di Monte S. Angelo, Via Cintia, 80126 Napoli, Italy
Francesco Graffitti  Institute of Photonics and Quantum Sciences, School of
Engineering and Physical Sciences, Heriot-Watt University, Edinburgh EH14 4AS,
UK  Nijil Lal  Dipartimento di Fisica, Università di Napoli Federico II,
Complesso Universitario di Monte S. Angelo, Via Cintia, 80126 Napoli, Italy
Andrea Chiuri  Enea - Centro Ricerche Frascati, via E. Fermi 45, 00044
Frascati, Italy  Simone Santoro  Enea - Centro Ricerche Frascati, via E. Fermi
45, 00044 Frascati, Italy  Luigi Santamaria Amato  Italian Space Agency (ASI),
Centro di Geodesia Spaziale ‘Giuseppe Colombo’, Località Terlecchia, 75100
Matera, Italy  Lorenzo Marrucci  Dipartimento di Fisica, Università di Napoli
Federico II, Complesso Universitario di Monte S. Angelo, Via Cintia, 80126
Napoli, Italy  Corrado de Lisio  Dipartimento di Fisica, Università di Napoli
Federico II, Complesso Universitario di Monte S. Angelo, Via Cintia, 80126
Napoli, Italy  Vincenzo D’Ambrosio<EMAIL_ADDRESS>Dipartimento
di Fisica, Università di Napoli Federico II, Complesso Universitario di Monte
S. Angelo, Via Cintia, 80126 Napoli, Italy

###### Abstract

Quantum interference is a central resource in many quantum-enhanced tasks,
from computation to communication protocols. While it usually occurs between
identical input photons, quantum interference can be enabled by projecting the
quantum state onto ambiguous properties that render the photons
indistinguishable, a process known as a quantum erasing. Structured light, on
the other hand, is another hallmark of photonics: it is achieved by
manipulating the degrees of freedom of light at the most basic level and
enables a multitude of applications in both classical and quantum regimes. By
combining these ideas, here we design and experimentally demonstrate a simple
and robust scheme that tailors quantum interference to engineer photonic
states with spatially structured coalescence along the transverse profile, a
type of quantum mode with no classical counterpart. To achieve this, we
locally tune the distinguishability of a photon pair via spatial structuring
of their polarisation, creating a structured quantum eraser. We believe these
spatially-engineered multi-photon quantum states may be of significance in
fields such as quantum metrology, microscopy, and communications.

##  I Introduction

In the Hong-Ou-Mandel (HOM) effect, quantum interference occurs when two
indistinguishable photons, entering a beamsplitter (BS), take the same output
path (photon bunching) [1]. Although the HOM effect is typically investigated
by tuning the temporal matching between two identical photons, the
indistinguishability required for the phenomenon makes it fundamentally
sensitive to all degrees of freedom (DoFs), from polarisation [2, 3] to
frequency [4, 5], and time-bins [6] as well as for collinear spatial modes
[7], including multi-particle [8, 9] and high dimensional scenarios [10, 11,
12, 13]. Monitoring this effect has rendered it a versatile tool with innate
sensitivity to measure a wide array of variations between the two inputs for
quantum enhanced measurements [14, 3, 15]. With the advent of sensitive
cameras enabling spatially resolved observation [16], measurements across the
spatial DoF [17, 18] have been made possible, such as dip tracking for
spatially-resolved height measurements [19] or insight into the spatial-
temporal [20] effects from multimode spontaneous parametric down-conversion
(SPDC).

The fundamental quantum nature of HOM interference also lends itself to tests
of quantum mechanics [21, 22], among which a notable example is the quantum
eraser [2, 23]. Such a paradigm allows one to restore quantum interference,
even if two photons are made distinguishable before entering the BS by
‘marking’ a degree of freedom such as polarisation. This is achieved by
projecting the outputs of both exit ports onto a basis that cannot yield
information on which path the photon took through the BS and so making the
photons effectively indistinguishable again. Moreover, the projection process
can allow one to edit the state by introducing phases to vary bunching
behaviour to anti-bunching and vice versa. One may accordingly imagine
engineering a desired photon number in an augmented state, lending itself to
be used as a structuring tool.

Structured light, in general, is a very powerful concept in modern optics,
spanning from the classical regime to fundamental quanta [24]. For instance,
classical implementations exhibit enhanced sensing [25, 26, 27, 28, 29],
microscopy [30] and communication [31] capabilities, while quantum
mechanically, it provides a test-bed for quantum mechanics [32, 33], secure
high-dimensional communication [34] and increased resilience to noise [35, 36,
37]. As such, there is a strong interest in tailoring complex or new
structures, such as designing across non-local degrees of freedom for quantum
skyrmions [38] or harnessing non-homogeneity for complex entanglement
structures [39, 40, 41]. It follows that, by designing a mode with structured
photon coalescence, we can combine the advantages of both concepts, holding
the possibility for direct impact in applications based on quantum
interference or structured light as well as unlocking new prospects.

By altering a given DoF before the BS in HOM interference, we already find
vanilla photon number engineering occurs by default, where bunching forms NOON
states [42, 43, 44] and, similarly, anti-bunching engineers non-local
entangled states [45]. Moreover, coincidence detection acts as a filter for
particular Bell and high-dimensional spatial states [10], while altering the
modal distinguishability between photons, facilitates space-time entanglement
engineering [46, 47] or the population of spatial modes [7, 12]. Such state
engineering can be extended to photon subtraction or additions with continuous
variables [48]. Where, in these cases, interference was used to postselect or
allocate photons to particular spatial modes, here we demonstrate a scheme to
spatially structure the quantum erasing process itself, thus obtaining a
photonic state with no classical counterpart, that is directly structured in a
quantum property of the field, i.e., photonic coalescence.

To do this, we harness the innate dependency of the HOM effect on identical
conditions to demonstrate how spatially tailoring a degree of freedom can
allow one to spatially tailor quantum interference. We achieve this by
exploiting geometric phase devices to generate non-uniform structure in the
polarisation degree of freedom and, having done so, a spatially-varying
structure in the distinguishability for the input photons. We therefore
implement a quantum eraser [2] via polarisation projective measurements in
order to study the caveats associated with these engineered states and elicit
conditions where coalescent structures can be heralded or erased. We
subsequently provide a framework for how one may engineer the bunching and
anti-bunching distribution across the transverse mode profile by exploiting
the simple generation of vector vortex (VV) modes. This approach and
subsequent structures, however, may be generalised to achieve arbitrary
freedom in engineering this fundamentally quantum property towards developing
a unique and versatile tool for applications in fields such as quantum
metrology, microscopy and communications.

Figure 1: Spatially engineering quantum interference. (a) In a quantum eraser,
two photons in different polarisation states are quantum interfering when
their which-path information is erased through two polarisers. Depending on
the polariser orientation, it is possible to tune photon coalescence from
bunching to antibunching. This is reflected in a HOM curve (when using bucket
detection of coincidences) that shows a dip or peak when the photons are
temporally indistinguishable (in), relative to the coincidence rate measured
when temporally distinguishable (out). (b) The polarisation distribution of
input photons can be spatially tailored to engineer local variations in the
quantum interference when overlapped on a 50:50 BS. The interference space-
variant behaviour is fully illustrated for three representative specific
locations, depicted using distinct colours (blue, green, pink). When tuning
the temporal distinguishability, the polarisation mixing in the quantum eraser
results in a corresponding space-variant distribution of HOM, exhibiting
peaks, dips or flat curves. This takes place despite one photon being detected
with a bucket detector.

##  II Results

Concept. The concept behind structuring quantum interference for tailoring
photon coalescence is illustrated in Figure 1. Although typical HOM scenarios
imply identical photons, quantum interference can be obtained also when the
two photons entering the BS are distinguishable, thanks to the quantum erasing
process [22]. When the which-path information is encoded in polarisation, for
instance, quantum interference can be enabled by placing a polariser on each
of the output paths, thus performing two projective measurements that erase
the which-path information. Interestingly, depending on the initial
polarisation state and each polariser’s orientation, it is possible to fully
tune photon coalescence from bunching to antibunching [Fig. 1(a)] [2]. We
could therefore exploit this feature in order to generate a structured quantum
state with a tailored coalescence in the transverse plane if we design a
spatially dependent quantum eraser. To this end, we consider a scenario where
the polarisation of each, otherwise indistinguishable photon, is given a non-
uniform distribution in the transverse plane before impinging on separate
ports of a 50:50 BS. A polariser is moreover placed in each of the output
paths of the BS. By tailoring the polarisation profile of each photon and
selecting the polariser’s orientation, we can have full control over the
spatial distribution of the photonic coalescence [Fig. 1(b)]. In other words,
by structuring quantum interference, photon bunching is tailored along the
optical mode transverse profile so that in each position we can have a zero-
or-two photon NOON state (HOM dips), a single photon (HOM peak), or some
combination of the two.

Figure 2: Experimental setup. After the spatial filtering with single mode
fibers, two photons ($\lambda=810$ nm), one in each path, are sent into two
different input ports of a 50:50 beamsplitter (BS) as in a typical HOM
setting. However, before reaching the BS, each of the photons is prepared in
the desired vector vortex (VV) mode (with spatial profiles shown as insets) by
sending horizontally polarised light through q-plates (QP) with $q=0.5$.
Placement of an additional half-waveplate (HWP) after QP1 transformed one of
the photons to the $\pi$ mode. The temporal delay between the two photons was
controlled by a translation stage mounted in the arm containing QP2. To
realise bucket detection, the output photons were coupled (D1,D2) into
multimode fibres and sent to two avalanche photodiodes (APD) connected to a
coincidence counter (CC). For spatially resolved detection, one output was
redirected via a flip mirror (FM) and focused with a lens onto an intensified
charge coupled device (Camera), and gated upon detection of the other photon
(D2). Which-way projections were performed by rotating linear polarisers
$P_{1}$ and $P_{2}$.

Experimental Implementation. We experimentally demonstrate this concept by
using the setup shown in Fig. 2, with full details provided in Methods.
Initially, SPDC biphotons ($\lambda=810$ nm) are separated in path and
spatially filtered with a single mode fiber. A set of polarisation correcting
waveplates (not shown in Fig. 2) facilitates uniform control of each photon’s
polarisation before being directed to the input ports of a 50:50 BS for
interference, giving the two-photon state
$\ket{\psi}_{in}=\left[\hat{a}_{A,m}^{{\dagger}}(\tau)\hat{a}_{B,n}^{{\dagger}}(\tau^{\prime})\right]\ket{0},\quad$
where $\hat{a}^{{\dagger}}$ is the creation operator for the respective modes,
($n$,$m$), with the time-bins $\tau$ and $\tau^{\prime}$, in the discrete
paths $\\{A,B\\}$ of each input port. A motorised path delay is placed in one
arm for tuning the temporal matching ($\tau^{\prime}\rightarrow\tau$) of the
photons. Initial matching of the input polarisation replicates the standard
HOM experiment when tuning the delay. Further insertion of geometric phase
elements, known as q-plates (QP) [49], before the BS allows the controlled
fashioning of spatially varying polarisation in each arm and the tuning of
each photon’s incident polarisation state, dependent on the topological charge
(q) of the element. This allows one to prepare a transverse mode ($n$) of the
form,

$\mathbf{E}_{n}(\mathbf{r},t)=\mathbf{e}_{s}(\varphi)f(r,t-\tau)e^{-i\omega
t},$ (1)

for each photon, where $f$ indicates the amplitude profile generally dependent
on the radial ($r$) coordinate, $\\{\mathbf{e}_{s}(\varphi)\\}$ are the
transverse unit vectors specifying the local mode polarisation as a function
of the azimuthal angle $\varphi$, $\omega$ is the mean (carrier) frequency of
the mode and time is $t$.

In particular, without loss of generality, we consider two modes that present
the same polarisation in some regions of the transverse profile, as depicted
in Fig. 1(b). As such, we choose the radial ($q=0.5$) and $\pi$ [$q=0.5$
followed by a half-waveplate (HWP)] modes depicted there, which are two first
order orthogonal VV modes [39], to be the input states of the BS. These may
respectively be described by the azimuthally dependent polarisations,

$\displaystyle\mathbf{e}_{rad}(\varphi)=\mathbf{e}_{H}\cos{\varphi}+\mathbf{e}_{V}\sin{\varphi}$
(2)
$\displaystyle\mathbf{e}_{\pi}(\varphi)=\mathbf{e}_{H}\cos{\varphi}-\mathbf{e}_{V}\sin{\varphi},$

where $e_{H}$ and $e_{V}$ are, respectively, the horizontal and vertical
polarisation unit-vectors.

As in typical HOM experiments, the interference or coalescent behaviour of our
modes are observed through the two-fold coincidences measured on the state

$\ket{\psi}_{out}=\frac{1}{2}\left[\hat{a}_{A,rad}^{{\dagger}}(\tau)\hat{a}_{B,\pi}^{{\dagger}}(\tau^{\prime})-\hat{a}_{A,\pi}^{{\dagger}}(\tau^{\prime})\hat{a}_{B,rad}^{{\dagger}}(\tau)\right]\ket{0},$
(3)

corresponding to the projection of the full BS-output state onto the subspace
in which the two photons are separated in ports A and B, and may be calculated
by the fourth-order correlation function [50], as detailed in the
Supplementary Information (SI). With this, we find the spatially varying
coincidence probabilities for arbitrary polarisation projections
{$\alpha$,$\beta$} on the photons detected in each port (respectively
corresponding to coordinates $\varphi_{1}$ and $\varphi_{2}$) when temporally
tuned ($\tau=\tau^{\prime}$)

$\centering\begin{split}C_{(In)}^{\alpha,\beta}(\varphi_{1};\varphi_{2})=\\\
&\frac{1}{4}\left|\left[\mathbf{u}_{\alpha}\cdot\mathbf{e}_{rad}(\varphi_{1})\right]\left[\mathbf{u}_{\beta}\cdot\mathbf{e}_{\pi}(\varphi_{2})\right]-\right.\\\
&\left.\left[\mathbf{u}_{\alpha}\cdot\mathbf{e}_{\pi}(\varphi_{1})\right]\left[\mathbf{u}_{\beta}\cdot\mathbf{e}_{rad}(\varphi_{2})\right]\right|^{2}\quad,\end{split}\@add@centering$
(4)

and temporally distinguishable ($\tau\neq\tau^{\prime}$)

$\centering\begin{split}C_{(Out)}^{\alpha,\beta}&(\varphi_{1};\varphi_{2})=\\\
&\frac{1}{4}\\{\left|\left[\mathbf{u}_{\alpha}\cdot\mathbf{e}_{rad}(\varphi_{1})\right]\left[\mathbf{u}_{\beta}\cdot\mathbf{e}_{\pi}(\varphi_{2})\right]\right|^{2}+\\\
&\left|\left[\mathbf{u}_{\alpha}\cdot\mathbf{e}_{\pi}(\varphi_{1})\right]\left[\mathbf{u}_{\beta}\cdot\mathbf{e}_{rad}(\varphi_{2})\right]\right|^{2}\\}\quad.\end{split}\@add@centering$
(5)

Here $\mathbf{u}_{\alpha}$ and $\mathbf{u}_{\beta}$ are the unit-vectors
directed along the detection polariser axes. For simplicity, we ignore here
the radial distribution as the polarisation variation depends only on
$\varphi$ (see SI for a more general treatment). Experimentally, we realise
these polarisation state projections by respectively placing polarisers
$P_{1}$ and $P_{2}$ in each output port of the BS.

As depicted in Figure 2, we adopted two strategies to observe the generated
structure by means of these coincidences, both in and out of the temporal
tuning. In one case, the photons from each output port were collected by
coupling into multimode fibres connected to bucket detectors placed in each
arm (flip mirror down), rendering no resolution of the spatial distribution.
This corresponds to measuring the correlations given in Eqs. (4) and (5) after
integrating over both azimuthal angles $\varphi_{1}$ and $\varphi_{2}$. In the
second case (flip mirror up), we performed spatially-resolved measurements of
one photon by replacing one bucket detector with a camera, which was then
conditioned on the spatially-unresolved detection of another photon by the
bucket detector in the other arm. This corresponds to measuring the
correlations given in Eqs. (4) and (5) after integrating over one angle only,
e.g. $\varphi_{2}$.

Figure 3:  Spatially tailored photon distributions from quantum interference.
Table of experimental outcomes for different choices of the polarisation
projections P1 (black arrows, row-wise) and P2 (black arrows, column-wise).
Hong-Ou-Mandel coincidence counts measured without spatial resolution, varying
the temporal delay, are shown as blue points in the graphs in the upper
section of each panel (solid curves are corresponding best fits). Error bars
from Poissonian statistics are too small to observe. The related spatially-
resolved coincidence images (spanning $\sim$ 60 X 60 pixels) in and out of the
temporal distinguishability, are given as insets to the HOM curves with arrows
denoting the measured distribution for the temporally tuned ($\Delta t=0$,
‘In’) case to the right and temporally distinguishable ($\Delta t<<0$, ‘Out’)
case to the left. The related spatially resolved visibilities are given as the
transverse distributions on the bottom, using the false-color scale shown on
the right-side bar. Analytically calculated visibility distributions are given
as top-right circular insets.

It is now possible to identify polarisation projection measurements that erase
the which-path information and enable quantum interference between these
structured modes, even for bucket detection. Intuitively, one may consider the
VV spatial distributions depicted in Fig. 2 which shows they share the same
polarisation state along the horizontal (H) and vertical (V) axes, while being
completely orthogonal along the diagonal (D) and anti-diagonal (A) directions.
It thus follows that a projection along H and V should destroy the which-path
information for the latter pairing and recover HOM interference. To observe
the related coalescence for such polarisations ($\alpha$ and $\beta$) we
define the visibility,
$\mathcal{V}_{\alpha,\beta}=(C^{\alpha,\beta}_{out}-C^{\alpha,\beta}_{in})/C^{\alpha,\beta}_{out}\in[-1,1]$
between the coincidences detected out ($C_{out}$) and in ($C_{in}$) the
temporal indistinguishably criteria [as depicted in Fig. 1 (b)]. This allows
us to easily recognise bunching as $\mathcal{V}$ becomes positive from anti-
bunching where $\mathcal{V}$ becomes negative with extreme cases of perfect
bunching (antibunching) being detected when $\mathcal{V}=1$ $(-1)$.

The graphs given in each panel of Figure 3 show the experimental outcomes of
the bucket-only detection as the delay between the two input photons was
varied by moving the delay line. Here, projections $(P_{1},P_{2})=$ (H,H)
elicit a dip at $\Delta t=0$, coinciding with a erasure of the which-way
information for the two photons and thereby restoring quantum interference.
For $(P_{1},P_{2})=$ (H,V), we find a peak instead of a dip, corresponding to
photon antibunching. [2, 11] Alternatively, cases $(P_{1},P_{2})=$
(H,A),(H,D),(A,D) and (A,A) result in a flat curve (no HOM dip nor peak).

On the other hand, for spatially-resolvable measurements (using the camera),
we expect the following spatially structured visibilities, corresponding to a
set of polarisation projections of $P_{1}\in\\{H,A\\}$ in the camera arm and
$P_{2}\in\\{H,V,D,A\\}$ in the bucket detector arm as depicted by the arrows
in Fig. 3:

$\displaystyle\mathcal{V}_{HH}=1$ (6) $\displaystyle\mathcal{V}_{HV}=-1$
$\displaystyle\mathcal{V}_{AH}=\cos{(2\varphi_{1})}$
$\displaystyle\mathcal{V}_{AV}=-\cos{(2\varphi_{1})}$
$\displaystyle\mathcal{V}_{HA}=\mathcal{V}_{HD}=\mathcal{V}_{AA}=\mathcal{V}_{AD}=0,$

obtained by integrating (4) and (5) over $\varphi_{2}$ due to bucket detection
in arm 2. In this case, images conditioned on coincidence with the bucket
detector were captured “out” (upper left inset of the curves) and “in” (upper
right inset of the curve) the temporally matched condition. Some asymmetry
towards the left in the distributions may be noted as the result of
experimental misalignment in the delay line to the camera (see Methods) as
well as the $q$-plate with respect to the photon distribution. This, however,
does not significantly detract from the expected distributions. Each measured
pair of coincidence distributions then allowed us to reconstruct the
visibility map (lower row) for the projection settings.

The resulting outcomes relate well to what was observed in the case of no
spatial resolution (bucket detection) as the coincidences measured by bucket
detection are proportional to the integrated pixel distribution over the mode
profile. We additionally compare these to the calculated profiles from Eq. 6
that are shown as upper-right insets of the visibility maps. Accordingly,
$(P_{1},P_{2})=$ (H,H) shows uniform coalescence across the spatial
distribution, while (H,V) exhibits a uniform anti-coalescence, within the
expectations of the calculated distributions. The cases of HA, HD, AA and AD
similarly yield a uniform structure in line with expectations of a zero
visibility across the measured profile.

Crucially, projections AH and AV, reveal spatially varying features with two
complementary lobes, showing coalescence and anti-coalescence across the
structure as predicted by Eqs. 6. These structures are not detectable without
spatial resolution, as shown by the flat curves in the corresponding graphs.
As a result, we herald a mode of structured bosonic coalescence with some
parts containing either zero or two photons as in the case of NOON states. At
the same time, other regions of the mode, corresponding to anti-coalescence,
will always be populated by one photon.

Discussion. With these results, we find interesting features, most notable are
the asymmetries in projected states made in either arm. For instance, a
diagonal projection only reveals structurally varying coalescence when placed
before the camera and one only observes total non-zero visibilities for H and
V polarisations projections. To understand this, one must consider, not just
the polarisation, but the paired spatial structures associated with them as
well as the global distinguishability. Here, projection on H and V for both VV
modes results in identical spatial lobes that overlap for both distributions,
thus rendering the photons indistinguishable, both globally and locally. For A
and D, however, the lobes, while partially overlapping, are orthogonal
structures and thus globally distinguishable. It follows that if one were to
integrate over the first case, the structure is indistinguishable and thus
facilitates in erasing the which-way information, while in the second case the
distinguishability is held. As the photons must be indistinguishable in both
arms to erase the which-path information and observe interference, the
perceived distinguishability for both detectors determines what is observed.
Bucket detection with A and D projections thus remains distinguishable,
leading to no visibility, even with an indistinguishable projection on the
camera. Conversely, bucket detection with H and V allows the locally
indistinguishable structures for A and D to be seen, despite the global
distinguishability. Additionally, a distinct reversal of the coalescent and
anticoalescent distributions can be seen for complementary polarisation
projections when structural coalescence or anticoalescence is present (i.e.
$P_{2}=$ H vs V for $P_{1}=$ H and A). One can thus locally switch between
bosonic and fermionic behaviour, or equivalently, between two/zero (NOON) and
single photon occupation. It follows that tailoring the quantum state can be
conveniently achieved by changing the polarisation distribution of the input
beams as well as the polarisation projections.

In conclusion, we have generated structured quantum states of light that have
no classical counterpart, with a spatially tailored bosonic coalescence. We
achieved this by exploiting a simple and robust scheme based on spatially
tailored distinguishability between two photons in a quantum eraser setting.
This additionally invites insight into the structured state where, in
particular, the use of orthogonal modes for each interfered photon showed a
global versus local HOM effect [6] such that detection without spatial
resolution yielded no sign of interference, but spatially resolved
measurements reveals its presence. Although these results were demonstrated
for VV modes, our technique may be generalised to intelligently engineer
arbitrary desired interference structures for varying purposes. Additionally,
we point out that programmable approaches, for tailoring the input
polarisation distributions, [51] would allow dynamic on-demand variations of
these states. As distinguishabilty is the property one needs to tailor, one
may extend our result to other degrees of freedom, such as temporally
structured light [52]. By nature of the ability to spatially tailor the
coalescent behaviour of photons, we believe these new states and engineering
thereof could be beneficial for fields such as quantum microscopy, metrology
and communications, including high-dimensional protocols.

##  III Methods

Experimental details. The two ($\lambda=810$ nm) signal-idler photons used in
the experiment (Fig. 2) for interference on the BS were generated in the state
$\ket{\psi}=\ket{H}\ket{V}$ via type-II degenerate collinear SPDC from a 30 mm
long ppKTP crystal pumped with a $\lambda=405$ nm continuous wave laser and
temperature phase-matched at $\sim$ 38 ∘C. They were then path separated with
a PBS and each spatially filtered by coupling into single mode fibers (SMF)
which directed the photons through a half-waveplate and quater-waveplate (for
preparation into the horizontal polarisation state) as well as electronically
tuned q-plates with the topological charge $q=0.5$ for polarisation-dependent
spatial structuring, before impinging onto a 50:50 BS for interference. One
SMF was placed on a motorised translation stage to introduce a path-delay for
temporally tuning the distinguishability. The output of each BS port was then
directed towards detectors. We switched between space-integrated and space-
resolved measurements by using a flip mirror. The first case (flip mirror
down) coupled the structured photons in each arm into multimode fibres for
detection by single-photon counting modules (APD) connected to a coincidence
counter with an integration window of 2 ns. The second case saw the photons in
one arm diverted by the flip mirror to a 30 m delay line for compensation of
the electronic delay associated with the triggering of the camera, before
being focused by a $f=150$ mm lens onto an intensified CCD. This camera was
then gated by the trigger provided from the APD, positioned in the other arm,
so as to spatially capture the photons in coincidence. Polarisation
projections on the structured states were made by inserting linear polarisers
into each output arm before the relevant detectors.

Vector vortex modes. In the experiment, VV modes were generated by exploiting
spin-to-orbital angular momentum conversion in a birefringent liquid crystal
slab with uniform retardation and an azimuthally varying optical axis, known
as a q-plate (QP). This device is chatacterised by a topological charge, $q$,
related to the number of rotations of the local optical axis and, accordingly,
the polarisation-dependent geometric phase imparted to incident light. In
Jones matrix formalism, the QP operation in the linear basis, where
$\ket{H}=[1;0]^{T}$ and $\ket{V}=[0;1]^{T}$ are horizontal and vertical
polarisation states, is described as,

$QP=\begin{bmatrix}\cos{}(2q\varphi)&\sin{}(2q\varphi)\\\
\sin{}(2q\varphi)&-\cos{(2q\varphi)}\end{bmatrix}.$ (7)

From this, it is straightforward to see the spatially-varying polarisation
distribution induced and dependence on polarisation. Taking a horizontally
polarised state as in the experiment, one obtains the following output:

$\displaystyle QP\ket{H}$
$\displaystyle=\begin{bmatrix}\cos{}(2q\varphi)&\sin{}(2q\varphi)\\\
\sin{}(2q\varphi)&-\cos{}(2q\varphi)\end{bmatrix}\begin{bmatrix}1\\\
0\end{bmatrix}=\begin{bmatrix}\cos{}(2q\varphi)\\\
\sin{}(2q\varphi)\end{bmatrix}$ (8)

and with $q=0.5$ as in the experiment, the state described in Eq. 2 is
obtained. By adding a HWP = $[1,0;0,-1]^{T}$ where the optical axis aligned
horizontally with the QP, the following transformation is achieved:

$\displaystyle[QP]$ $\displaystyle=\begin{bmatrix}1&0\\\
0&-1\end{bmatrix}\begin{bmatrix}\cos{}(2q\varphi)&\sin{}(2q\varphi)\\\
\sin{}(2q\varphi)&-\cos{}(2q\varphi)\end{bmatrix}$ (9)
$\displaystyle=\begin{bmatrix}\cos{}(2q\varphi)&\sin{}(2q\varphi)\\\
-\sin{}(2q\varphi)&\cos{}(2q\varphi)\end{bmatrix}$

that, acting on the state $\ket{H}$, produces the desired state
$\ket{\pi}=\cos{(\varphi)}\ket{H}-\sin{(\varphi)}\ket{V}$ for $q=0.5$.

## Acknowledgements

We acknowledge support from the Italian Ministry of Research (MUR) through the
PRIN 2017 project “Interacting photons in polariton circuits” (INPhoPOL) and
the PNRR project PE0000023-NQSTI. We also acknowledge support from NATO
through SPS Project HADES - MYP G5839 and from the Italian Space Agency (ASI)
through the ”High dimensional quantum information” (HDQI) project.

## References

  * [1] Hong, C. K., Ou, Z. Y. & Mandel, L.  Measurement of subpicosecond time intervals between two photons by interference.  _Physical Review Letters_ 59, 2044–2046 (1987).
  * [2] Kwiat, P. G., Steinberg, A. M. & Chiao, R. Y.  Observation of a “quantum eraser”: A revival of coherence in a two-photon interference experiment.  _Physical Review A_ 45, 7729 (1992).
  * [3] Harnchaiwat, N., Zhu, F., Westerberg, N., Gauger, E. & Leach, J.  Tracking the polarisation state of light via hong-ou-mandel interferometry.  _Optics express_ 28, 2210–2220 (2020).
  * [4] Khodadad Kashi, A. & Kues, M.  Spectral hong–ou–mandel interference between independently generated single photons for scalable frequency-domain quantum processing.  _Laser & Photonics Reviews_ 15, 2000464 (2021).
  * [5] Hong, L., Chen, Y. & Chen, L.  Delayed-choice quantum erasure with nonlocal temporal double-slit interference.  _New Journal of Physics_ 25, 053014 (2023).
  * [6] Nitsche, T. _et al._ Local versus global two-photon interference in quantum networks.  _Physical Review Letters_ 125, 213604 (2020).
  * [7] D’Ambrosio, V., Carvacho, G., Agresti, I., Marrucci, L. & Sciarrino, F.  Tunable two-photon quantum interference of structured light.  _Physical Review Letters_ 122, 013601 (2019).
  * [8] Agne, S. _et al._ Observation of genuine three-photon interference.  _Physical Review Letters_ 118, 153602 (2017).
  * [9] Menssen, A. J. _et al._ Distinguishability and many-particle interference.  _Physical Review Letters_ 118, 153603 (2017).
  * [10] Zhang, Y. _et al._ Engineering two-photon high-dimensional states through quantum interference.  _Science Advances_ 2, e1501165 (2016).
  * [11] Walborn, S. P., de Oliveira, A. N., Pádua, S. & Monken, C. H.  Multimode hong-ou-mandel interference.  _Physical Review Letters_ 90, 143601 (2003).
  * [12] Hiekkamäki, M. & Fickler, R.  High-dimensional two-photon interference effects in spatial modes.  _Physical Review Letters_ 126, 123601 (2021).
  * [13] Liu, Z.-F. _et al._ Hong-ou-mandel interference between two hyperentangled photons enables observation of symmetric and antisymmetric particle exchange phases.  _Physical Review Letters_ 129, 263602 (2022).
  * [14] Lyons, A. _et al._ Attosecond-resolution hong-ou-mandel interferometry.  _Science advances_ 4, eaap9416 (2018).
  * [15] Triana-Arango, F., Ramos-Ortiz, G. & Ramírez-Alarcón, R.  Spectral considerations of entangled two-photon absorption effects in hong–ou–mandel interference experiments.  _The Journal of Physical Chemistry A_ 127, 2608–2617 (2023).
  * [16] Jachura, M. & Chrapkiewicz, R.  Shot-by-shot imaging of hong-ou-mandel interference with an intensified scmos camera.  _Optics Letters_ 40, 1540–1543 (2015).
  * [17] Chrapkiewicz, R., Jachura, M., Banaszek, K. & Wasilewski, W.  Hologram of a single photon.  _Nature Photonics_ 10, 576–579 (2016).
  * [18] Lipka, M. & Parniak, M.  Single-photon hologram of a zero-area pulse.  _Physical Review Letters_ 127, 163601 (2021).
  * [19] Ndagano, B. _et al._ Quantum microscopy based on hong–ou–mandel interference.  _Nature Photonics_ 16, 384–389 (2022).
  * [20] Devaux, F., Mosset, A., Moreau, P.-A. & Lantz, E.  Imaging spatiotemporal hong-ou-mandel interference of biphoton states of extremely high schmidt number.  _Physical Review X_ 10, 031031 (2020).
  * [21] Torgerson, J., Branning, D., Monken, C. & Mandel, L.  Violations of locality in polarization-correlation measurements with phase shifters.  _Physical Review A_ 51, 4400 (1995).
  * [22] Pan, J.-W. _et al._ Multiphoton entanglement and interferometry.  _Rev. Mod. Phys._ 84, 777–838 (2012).
  * [23] Rezai, M., Wrachtrup, J. & Gerhardt, I.  Coherence properties of molecular single photons for quantum networks.  _Physical Review X_ 8, 031026 (2018).
  * [24] Rubinsztein-Dunlop, H. _et al._ Roadmap on structured light.  _Journal of Optics_ 19, 013001 (2016).
  * [25] D’Ambrosio, V. _et al._ Photonic polarization gears for ultra-sensitive angular measurements.  _Nature Communications_ 4, 2432 (2013).
  * [26] Barboza, R. _et al._ Ultra-sensitive measurement of transverse displacements with linear photonic gears.  _Nature Communications_ 13, 1080 (2022).
  * [27] Yuan, G. H. & Zheludev, N. I.  Detecting nanometric displacements with optical ruler metrology.  _Science_ 364, 771–775 (2019).
  * [28] Bag, A., Neugebauer, M., Wozniak, M., Leuchs, G. & Banzer, P.  Transverse kerker scattering for angstrom localization of nanoparticles.  _Physical Review Letters_ 121, 193902 (2018).
  * [29] Tischler, N. _et al._ All-optical self-referenced transverse position sensing with subnanometer precision.  _ACS Photonics_ 5, 3628–3633 (2018).
  * [30] Maurer, C., Jesacher, A., Fürhapter, S., Bernet, S. & Ritsch-Marte, M.  Tailoring of arbitrary optical vector beams.  _New Journal of Physics_ 9, 78 (2007).
  * [31] Wang, J. _et al._ Terabit free-space data transmission employing orbital angular momentum multiplexing.  _Nature Photonics_ 6, 488–496 (2012).
  * [32] Nape, I., Ndagano, B. & Forbes, A.  Erasing the orbital angular momentum information of a photon.  _Physical Review A_ 95, 053859 (2017).
  * [33] D’Ambrosio, V. _et al._ Experimental implementation of a kochen-specker set of quantum tests.  _Physical Review X_ 3, 011012 (2013).
  * [34] Erhard, M., Krenn, M. & Zeilinger, A.  Advances in high-dimensional quantum entanglement.  _Nature Reviews Physics_ 2, 365–381 (2020).
  * [35] Ecker, S. _et al._ Overcoming noise in entanglement distribution.  _Physical Review X_ 9, 041042 (2019).
  * [36] Nape, I. _et al._ Revealing the invariance of vectorial structured light in complex media.  _Nature Photonics_ 16, 538–546 (2022).
  * [37] D’Ambrosio, V. _et al._ Complete experimental toolbox for alignment-free quantum communication.  _Nature Communications_ 3, 961 (2012).
  * [38] Ornelas, P., Nape, I., Koch, R. d. M. & Forbes, A.  Non-local skyrmions as topologically resilient quantum entangled states of light.  _arXiv preprint arXiv:2210.04690_ (2022).
  * [39] D’Ambrosio, V. _et al._ Entangled vector vortex beams.  _Physical ReviewA_ 94, 030304 (2016).
  * [40] Gao, X. _et al._ Full spatial characterization of entangled structured photons.  _arXiv preprint arXiv:2304.14280_ (2023).
  * [41] Graffitti, F. _et al._ Hyperentanglement in structured quantum light.  _Physical Review Research_ 2, 043350 (2020).
  * [42] Nagata, T., Okamoto, R., O’brien, J. L., Sasaki, K. & Takeuchi, S.  Beating the standard quantum limit with four-entangled photons.  _Science_ 316, 726–729 (2007).
  * [43] Walther, P. _et al._ De broglie wavelength of a non-local four-photon state.  _Nature_ 429, 158–161 (2004).
  * [44] Lee, S.-Y. & Nha, H.  Second-order superposition operations via hong-ou-mandel interference.  _Physical Review A_ 85, 043816 (2012).
  * [45] Chen, Y. _et al._ Temporal distinguishability in hong-ou-mandel interference for harnessing high-dimensional frequency entanglement.  _npj Quantum Information_ 7, 167 (2021).
  * [46] Branning, D., Grice, W. P., Erdmann, R. & Walmsley, I. A.  Engineering the indistinguishability and entanglement of two photons.  _Physical Review Letters_ 83, 955–958 (1999).
  * [47] Branning, D., Grice, W., Erdmann, R. & Walmsley, I.  Interferometric technique for engineering indistinguishability and entanglement of photon pairs.  _Physical Review A_ 62, 013814 (2000).
  * [48] Biagi, N., Francesconi, S., Zavatta, A. & Bellini, M.  Photon-by-photon quantum light state engineering.  _Progress in Quantum Electronics_ 100414 (2022).
  * [49] Marrucci, L., Manzo, C. & Paparo, D.  Optical spin-to-orbital angular momentum conversion in inhomogeneous anisotropic media.  _Physical Review Letters_ 96, 163905 (2006).
  * [50] C. Cohen-Tannoudji, G. G., J. Dupont-Roc.  _Photons and Atoms: Introduction to Quantum Electrodynamics_ (John Wiley & Sons, New York, 1997).
  * [51] Liu, Z. _et al._ Flexible method for generating arbitrary vector beams based on modified off-axis interference-type hologram encoding.  In _Photonics_ , vol. 9, 949 (2022).
  * [52] Specht, H. P. _et al._ Phase shaping of single-photon wave packets.  _Nature Photonics_ 3, 469–472 (2009).


# Deployment of Aerial Robots after a major fire of an industrial hall with
hazardous substances, a report

Hartmut Surmann1, Dominik Slomma1,3, Stefan Grobelny2, Robert Grafe3 *This
work was founded by the Federal Ministry of Education and Research (BMBF)
under grant number 13N14860 (A-DRZ https://rettungsrobotik.de/).1University of
Applied Science, Gelsenkirchen, 2Fire Department of Dortmund, Institute of
Fire Service and Rescue Technology, 3German Rescue Robotic Centre,
Dortmund.corresponding author<EMAIL_ADDRESS>material at:
https://github.com/RoblabWh/Berlin

###### Abstract

This technical report is about the mission and the experience gained during
the reconnaissance of an industrial hall with hazardous substances after a
major fire in Berlin. During this operation, only UAVs and cameras were used
to obtain information about the site and the building. First, a geo-referenced
3D model of the building was created in order to plan the entry into the hall.
Subsequently, the UAVs were used to fly in the heavily damaged interior and
take pictures from inside of the hall. A 360° camera mounted under the UAV was
used to collect images of the surrounding area especially from sections that
were difficult to fly into. Since the collected data set contained similar
images as well as blurred images, it was cleaned from non-optimal images using
visual SLAM, bundle adjustment and blur detection so that a 3D model and
overviews could be calculated. It was shown that the emergency services were
not able to extract the necessary information from the 3D model. Therefore, an
interactive panorama viewer with links to other 360° images was implemented
where the links to the other images depends on the semi dense point cloud and
located camera positions of the visual SLAM algorithm so that the emergency
forces could view the surroundings.

##  I Introduction

Young scientists in the field of robotics are often asked what their robot,
algorithm or sensor is good for and the answer is often for use in the field
of rescue robotics. On the one hand, the question in the field of research is
unfair, on the other hand, it also shows how little known requirements are in
the real use of robots in dangerous situations and that is exactly what we
want to improve with this article. In the evening on Thursday, 11.02.2021 a
fire in a metalworking factory in Berlin, Germany, could only be extinguished
after more than 12 hours of work [1]. Hazardous substances were released
during use. Due to the stored hazardous substances residents had to keep their
windows closed. A radius of about three kilometers around the fire was
affected. According to the fire brigade, 2,000 square meters of the
approximately 5,000 square meter burned initially. Despite the extinguishing
work, the fire then spread to the entire production hall. Up to 170
firefighters were on duty at the same time.

Due to the high level of damage, the hall was closed and an entry ban was
issued. So, how to get images and information from the inside?

##  II The Job

On February 22nd, 2021, the Berlin police submitted an administrative
assistance request to the Dortmund fire department for support with special
UAV technology as part of a fire investigation.

As a result, a team of emergency services from the Dortmund fire department
(FwDO) and staff members from the German Rescue Robotics Centre (DRZ) and
Westphalian University of Applied Science were put together and set off to
Berlin for a three-day mission with the robotic command vehicle (RobLW). The
order for the team was the digital representation of the outside and inside
area of the industrial hall. In addition to the UAV and image technology, the
RobLW was also used on site, with the built-in technology of which the
situation images of the UAVs were displayed in real time in order to be able
to create corresponding 3D views. Due to the remaining 10-30 cm high and
potentially contaminated extinguishing water on the ground, it was clear from
the outset that ground robots could not be used. The recordings of the image
and video material were made available to the Berlin police for assessing the
situation and for further investigations.

The use of robotic systems in this dangerous scenario for emergency services
was necessary because the dangers for them could not be conclusively assessed
from the outside. By supporting with UAV and special image technology, it was
possible for the first time to see the inside of the hall, because of flying
the inside of the hall with previous UAV technology by trained UAV pilots was
too risky. In addition to the provided 3D views, numerous findings for the
further development of robot technology were obtained.

##  III State of the art

Rescue robotics encompasses a broad field in research. Because the robots are
tested in optimal environments, they rarely find application in real-world
operations. Researching the results in real-world conditions is fundamentally
important to optimize the robots for specific scenarios. For example, ground
robots could not be optimally deployed in a collapsed building in Cologne
Germany in 2009, because the debris did not allow the robots to advance [2].
Two years later, in 2011, there were major earthquakes in Japan, whereupon
ground robots were used to investigate hazardous areas [3]. In addition to
ground robots, underwater robots were also used to find missing people and
explore other areas. In 2012 and 2016, there were strong earthquakes in Italy,
after which buildings were in danger of collapsing [4, 5, 6].There, ground and
aerial robots were used to investigate building damage. Since these robots
also could not penetrate all areas, water robots were used to identify the
remaining areas. These missions show the advantages of using robots and reduce
the risk of humans having to put themselves in danger. However, it is clear
from the missions that the connection of the research to the real missions is
dependent on the force used and how it is trained. Therefore, different real-
world environments are used to optimize the robots and train the responders
[7]. For example, the Fukushima site provides an optimal test site for testing
the use of robots [8]. In addition to the missions and test areas, it is
noticeable that the use of UAVs is fundamentally important in the rescue
sector. UAVs make it possible to observe a large area in a very short time and
quickly obtain an overall view [9, 10].

##  IV Deployment

###  IV-A Hardware and software equipment

####  IV-A1 Robotic Command Vehicle (RobLW)

The German Rescue Robotics Centre (DRZ) and the Dortmund Fire Department
(FwDO) cooperate in using the RobLW. The DRZ uses the vehicle primarily for
research, to analyse in which areas of operations the robots can support the
emergency forces in a meaningful way. By supporting aerial and ground robots,
it is possible to enter and investigate areas without putting people in
danger. In addition to research, the RobLW is used by the FwDO to test the
introduced findings in real-life scenarios. For this purpose, the DRZ works
closely with the Dortmund fire department. The RobLW is 7 m long, 2 m wide and
3 m high and can be divided into three areas.

Figure 1: Overview of the RobLW. Top: The RobLW. Bottom left: Trunk with a
ground robot. Bottom right: The command center with two monitor workstations.

The front area is the command compartment which is used to control the
vehicle. In addition, it is possible to contact the command centre and other
emergency units by radio from there. It is also possible to connect to the
vehicle’s network with a separate computer to share information or access the
vehicle’s infrastructure. In the rear area, various peripheral equipment and
components are stored for use. Furthermore, the compartment provides a storage
area for ground robots and UAVs to be transported. The middle area represents
the operation centre. This is where the robots and UAVs are controlled and
where the firefighters can be supported by suitable assistance systems. For
this purpose, the operation centre of the RobLW is equipped with hardware for
two seats and a powerful server. A screen with an Intel NUC and the associated
peripherals is available for each operator. The Intel NUCs are used
exclusively for the visualization of camera data or the visual representation
of, for example, 3D created environments. The calculations for the creation of
three-dimensional environments or artificial neural networks take place on the
server. This has an AMD Ryzen Threadripper processor with 64 cores, 64 GB RAM
and an Nvidia Geforce RTX 2080 with 8 GB dedicated memory. To ensure
communication between the server and the NUCs, a special TDT router is used.
This not only takes care of the communication, but also enables access to the
Internet via the mobile network. It offers the possibility to download local
maps of the environment during an operation. In addition, current data can be
sent to the operations control centre to get a better overview of the current
situation. Furthermore, the router creates its own Wi-Fi network. This offers
the option that the operation commander does not have to evaluate the data
directly in the RobLW but can be closer to the operation site if it is
connected to the RobLW. The operations centre of the RobLW also has its own
radios, for example, to coordinate with other UAV pilots or to provide the
operations commander with current details as quickly as possible.

Since the server is mainly responsible for processing the various data from
the robots, it contains the primary software to support the fire department.
To ensure that the operators know where the robots are at all times, the
server contains a so-called situation image system. This receives data from
the robots, such as the camera data and the GPS information of the individual
robots. This information is used to track the robots in a client software in
real time and to determine their position. Furthermore, the information of
each robot can be viewed and analysed. In addition to the positional image
system, the server has the WebODM 111Drone Mapping Software:
www.opendronemap.org/webodm/. This is a web-based tool that uses camera images
to create (offline) a three-dimensional representation of the environment. The
WebODM provides a scaled point cloud of the environment which can be used to
measure the area of operation or holes in a building at risk of collapse. This
offers the advantage that conclusions can be drawn and a better approach can
be planned [11].

Figure 2: RobLW infrastructure. The operator sends the information to RobLW,
where it is then processed on the RobLW server. Afterwards, the processed data
can be viewed in RobLW on the NUCs or this information can be sent to the
control station.

####  IV-A2 UAVs

Due to the holes in the hall ceiling caused by the fire, it was assumed that
it would be possible to fly UAVs into the building to get an overview of the
degree of destruction. A variety of UAVs were brought in for the mission. The
DJI Phantom 4 (P4P) was considered due to its maximum flight speed of 72 km/h
and stable flying in less than optimal weather. This UAV makes it possible to
fly over the entire area in a very short time to get an overview of the
current situation on site. However, since the P4P only has a maximum camera
resolution of 4000×3000 and a maximum video bit rate of 60 Mbps, the DJI Mavic
Pro 2 (MP2) was taken along for detailed recordings. This UAV has a camera
resolution of 5472×3648 and a maximum video bit rate of 100 Mbps. Both UAVs
can capture 360° panorama images autonomously.

The two UAVs are relative large with a diagonal of 60cm, which is why the DJI
Spark and the DJI Mavic Mini were taken along for narrower indoor spaces.
These small UAVs were to be used when the destroyed hall had many cables or
other broken objects hanging from the ceiling, making it impossible to fly
safely in the hall with the MP2 and the P4P. The Spark can take panorama
images but not the Mavic Mini.

####  IV-A3 Cameras

Panoramas provide the most information in an image, so mounts have been
developed for the P4P and MP2. With these mounts it is possible to attach 360°
cameras to the UAVs. The cameras that were available are the Insta 360 One X
and the Insta 360 Evo. These have a video function that provides maximum
information of the environments through minimal movements of the UAVs. Both
cameras have a video resolution of 5760x2880 (15.8MP) at 30 fps.

###  IV-B Situation overview model

Since the hall represents unknown terrain, an overview was needed at the
beginning. For this purpose, a meander flight with the MP2 was made in a
height of 45 meter over the railing in which the hall is located. The UAV
flies over the area and creates orthogonal images from the air at regular
intervals. The spacing between the images can be adjusted by setting the
overlap of the images. In this deployment, an overlap of 70% was set,
generating 167 images from the UAV. These images were then processed using the
WebODM tool. WebODM uses Structure from Motion (SfM) and Multi View Stereo
(MVS) to create a scaled 3D point cloud that can be viewed in a web browser.
Figure 3 shows the point cloud as well as the camera positions, in blue
rectangles, over the terrain. By selecting a blue rectangles the image can be
displayed in its original resolution (top right). Through the scaled point
cloud, the openings of the burned roof hatches could be measured, which have a
width of 1m and a length of 2.3m.

Figure 3: Result of the first action is a georeferenced 3D point cloud. It is
calculated out of 167 images acquired in a 12 minutes flight of the Mavic at a
height of 45 meters. The calculation of the model in the RobLW needs 15
minutes. The measurements taken in the model are used to plan the second
action.

###  IV-C First flight in - Connection problems with the drone

From the openings measured above, it was obvious that flying through the
ceiling opening into the hall was possible with the P4P and the MP2. There are
two problems with this. The small aperture angle of the UAV camera (FOV about
80°) does not allow to see the boundaries of the opening while flying through
it. The second problem is to fly the UAV back out of the hall, because the UAV
cameras can be pointed forward and downward but not upward. To ensure this, we
took the ”two pilots approach”. While the first UAV enters the hall, the
second UAV is positioned exactly above the entry point and assists in flying
in. While the first UAV autonomously creates the panoramas in the hall, the
second UAV waits above the opening, and after completing the shots, the second
pilot navigates the first UAV back out through the opening. For the fly-in,
the MP2 was chosen due to the slightly better camera resolution and was
equipped with propeller guards. Upon approaching the openings, it was
noticeable that the metal of the roof hatches had been severely bent by the
fire. This significantly reduced the width of the holes and made it very
difficult to fly into the hall. Some openings, for example numbers 4 and 13,
could not be flown at all (Fig. 12). To test the ”two pilots approach”, a
particularly large opening was first selected and the two UAV pilots were
positioned next to the RobLW, right next to the hall. However, this meant that
the radio signal from the remote control had to pass through several layers of
concrete and steel, causing the radio link to the MP2 to constantly break
down. Normally, if the radio link was lost, the UAV would activate the ”Return
to Home” (RTH) function, which would cause the UAV to fly back to the launch
point on its own. However, the loss of radio link was anticipated, so the
safety function was disabled. Furthermore, it is difficult to generate
targeted photos without a radio connection. The UAV was able to fly in and out
again, but the attempt was aborted due to the difficulties.

###  IV-D Second flight in - More stable connection to the drone

The frequent disconnections caused an increased risk of crashing the MP2.
Therefore, efforts were made to minimize, if not eliminate, the disconnects.
The burned hall bordered on an neighboring hall that was accessible. It was
located higher than the burned roof which in addition to the visual support
from the P4P allowed the pilot to better assess the area. Figure 4 left shows
the manoeuvre of the two UAVs. During this flight manoeuvre it was noticeable
that the lower UAV, the MP2, became more unstable the closer the P4P came.
This could be attributed to the P4P’s propellers swirling the air too much and
causing turbulence for the MP2. In addition Figure 4 shows the two UAV pilots
in the upper right portion of the image and the live stream shown at RobLW in
the lower right portion of the image. The elevated position meant that the
radio signal no longer had to pass through several layers of concrete and
steel, which meant that the radio signal did not break off when flying into
the hall. This meant that the radio signal could be locked in at any time,
allowing panoramas to be created more quickly in the hall. When panorama
function is executed, the UAV creates autonomously 26 individual images (12
MP) of the surroundings, which are then converted to a 360-degree panorama
with a resolution of 32 MP (on board). Creating a panorama takes about 1
minute. By post-processing the 26 images at the RobLW, panoramas with 128 MP
could be generated. For safety reasons, the MP2 was flown only 1.5m into the
hall.

Figure 4: Positioning of the two UAVs. The lower one penetrates the damage
structure and takes 26 images. The UAVs are controlled by a team of operators
and live data is transmitted to the RobLW.

###  IV-E Use of panoramic camera - Gathering of missing data

Another trick was used to capture the positions in the hall that could not be
flown directly because the openings were too small. A panoramic camera, namely
the Insta 360 One x (110g, 15.8 MP), was attached to the P4P with a 1.5 m long
thin rope and inserted into the respective opening from above (Figure 5). With
the Insta 360 One X, it was possible to record 20 minutes of video footage to
view all parts of the site. Figure 6 shows the approached roof hatches into
which the camera was lowered with the P4P. The problem with this approach is
that the video contains a lot of motion blur due to the moving rope.
Furthermore, there are some overexposed and underexposed frames especially
when entering the hall from the outside and back. The video has about 36000
frames generated at 30 fps. Furthermore, consecutive frames often show the
same or very similar scenes. Therefore, relevant (good) images had to be
extracted from the video. The quality of the images is significantly worse
than the panoramas taken directly with the UAV (15.8 MP vs. 128 Mp). Related
to a field of view of 90° this is 2 MP vs. 16 MP.

Figure 5: 360° camera connected to the P4P with a thin rope while inspecting
the hall.

##  V Results

Figure 6: Overview image after the initial meander mapping flight. The numbers
indicate position of penetration points for the UAV (skylight windows) and
therefore insight panorama 360 degree images. Bottom right is the position of
the RobLW. The position of the operators were first at the RobLW and second at
the neighbouring roof starting directly at the right sight of the image. The
line at the right side is a wall which separates the halls. The part right
from the line was not effected by the fire.

As mentioned above, the video must be post-processed to make important
information visible.

###  V-A Dataset filtering by visual SLAM

To reduce the dataset to the most necessary images, the SLAM framework
OpenVSLAM [12] was used. This framework uses the ORB2-SLAM [13] algorithm and
supports equirectangular camera types. With OpenVSLAM each keyframe of the
video was saved. This ensures that these images contain representative
information of the environment. This method reduced the dataset from 36000
frames to a dataset of over 700 key frames. Furthermore a flight trajectory
could be generated and also the high quality panoramas could be localized.

###  V-B Dataset filtering by blur detection

During the key frame calculation of the video data set by visual SLAM, only
non-representative images and duplicate images are removed from the original
data set. Images with motion blur were not completely removed. Therefore, a
Laplace Blur Detection [14, 15, 16, 17] was used to remove blurred images from
the video as well. In this detection, the Laplace filter is applied to the
image and then the variance is determined. If the variance is below a certain
threshold, then the image should normally be discarded. Figure 7 serves as an
example. However, since the variance changes with the environment and thus the
threshold must be adjusted, a dynamic threshold was added to the detector from
[14]. This takes into account the images before and after the current image
$B_{n}$. Therefore the variances of $k$ images and the variance of the current
image are calculated. The mean value of these corresponds to the threshold. We
have chosen the value 20 for the parameter $k$. A problem with the procedure
is that at the beginning of the algorithm or at the first image no previous
images exist, the same problem exists at the end of the data set. Therefore,
the parameter $j$ with the value 0 was introduced, which ensures that non-
existing images are not included in the calculation.

$\displaystyle\frac{1}{(k*2)+1+j}\sum_{i=n-k}^{n+k}Var(B_{i})\text{ }|\text{
}j=\begin{cases}j+1&\text{if }i<0\\\ j+1&\text{if }i>\text{end}\end{cases}$
(1)

Figure 7: Example of blur detection. At the top right a sharp image with a
high variance and at the top left an blurred image with a lower variance.
Below the corresponding variance curve of the blur detection.

###  V-C 3D Viewer

With the blur detection the dataset was reduced to a little more than 200
images and processed with WebODM. WebODM uses OpenSfM [18] to create semi
dense point clouds, as shown in Figure 8. This gives an abstract overview of
the environment and also localizes the camera positions, which can be used in
a 2D viewer. The dense point cloud was created using OpenMVS [19] and provides
detailed representation of the environment (figure 9). This detailed
representation could be used as an overview to get an overall impression of
the situation.

Figure 8: Semi dense point cloud with localized image position calculated with
structure from motion or visual SLAM. Figure 9: Dense point cloud of the
structure calculated with multi view stereo.

###  V-D 360° panorama viewer

Based on the localization from OpenSfM, an interactive panorama viewer was
implemented. It uses the Pannelum web viewer javascript plugin
222pannellum.org/. With this it is possible to view 360° panoramas
interactively. The viewer was extended by the localization information
(reconstruction.json) from OpenSfM, so that it is possible to switch between
the individual panoramas. As shown in Figure 10, there are several color-coded
circles. These represent the localized panoramas. The more red a point is, the
closer the panorama is to the current position. If the panorama is further
away, it will be blue. By clicking into the circle the next panorama appears.
The presentation was inspired by google street view.

Figure 10: Panorama viewer. The user can rotate, zoom and navigate in the
images. The red to blue circles indicate further 3D panorama images. Large and
red mean near by images, small and blue mean far away images according to the
localization in the semi dense point cloud.

##  VI CONCLUSIONS

This article reports on the use of the UAV in Berlin in February 2021 to
inspect an industrial hall with hazardous substances that was in danger of
collapsing after a large fire. It has been shown that the use of the panorama
function of the UAV provides optimal information of the environment, but there
is always a great risk that the UAV collides with the environment. Therefore,
the use of a 360 degree camera, which is let into the hall, is also suitable.
This allows image information to be collected from different heights without
the risk of crashing. The disadvantage of the camera is the post-processing of
the data. These have to be freed from non-optimal image information in order
to create a suitable 3D model of the environment. It also turned out that 3D
models are interesting for the robots, but not for the emergency services.
They have difficulties to find their way in a 3D model and to extract the
necessary information from it. Therefore, the 3D model should only serve as a
rough overview and the implemented panorama viewer was used by the emergency
forces to obtain the necessary information from the environment. Finally,
thanks to the unknown reviewers whose questions we will try to answer below.

  1. 1.

After the deployment, how about the decontamination of the UAVs and Insta 360?
After the mission, the UAVs and the 360° cameras were carefully cleaned. As
expected, a final measurement showed no decontamination, as both the UAVs and
the camera had no direct contact with the environment.

  2. 2.

What information the panorama viewer can obtain and the 3D model cannot? Due
to the high resolution of the single panorama images they contain a lot more
details compared to the dense point clouds. The calculation of dense point
clouds by multi view stereo algorithm downscale the image i.e. by a number of
4. Furthermore, has the used PatchMatch Algorithm difficulties with non
textured surfaces. Through the panorama viewer it was possible to view
individual panoramas and through their high resolution it was possible to zoom
into interesting areas to get a detailed view of the individual areas as well
as directly move to other places.

  3. 3.

Which types of analyzed data would be finally useful for polices further
investigations and when was the timing of its provision? Both the point clouds
as well as the panorama images are useful and important. The dense point
clouds give a spatial impression of the operation and serve for the
localization of the panoramas as well as for the measurements. The panoramas
in turn contain the details for assessing the statics and geometry of the
building as well as the cause of the fire. The outdoor 3D overview model was
calculated at the side (in the RobLW) and provided 30 Minutes after the UAV
flight. It served for the planning of the further proceeding. The panoramas
were also shown directly after the flights during the mission. The 3D interior
model and the final browser-based panoramic viewer were delivered online 3
days after the mission.

  4. 4.

How the results helped for the further investigation? Due to the ban on
entering the hall after the fire, there was no information from the inside of
the hall, neither regarding the statics and danger of collapse, nor regarding
the cause of the fire. The images provided are the basis for pumping out the
decontaminated firefighting water, assessing the statics and planning the
deconstruction as well as determining the cause of the fire.

  5. 5.

Before the deployment, the UAV team had training for a similar application? If
so, what’s new in the real deployment comparing to the training? The UAV team
had already trained flying inside and outside buildings and flying into
buildings through windows. In addition, there was already operational
experience with flying robots during earthquakes, e.g. (Amatrice / Italy
2016). Furthermore, there was extensive experience in the development and
application of algorithms from the field of image processing and computer
science. Flying in through skylights instead of windows was new. Furthermore,
the images are new and different from those of earthquakes. Fire, especially
in industrial halls leads to massive destruction of the environment.

  6. 6.

Are there any improvements in the deployment of these drones? Not directly in
the use of UAVs, but in the development of the image processing and display
algorithms. This refers on the one hand to the better display (PanoViewer) and
on the other hand to the processing speed for data evaluation.

Figure 11: Detail view of the roof of the industrial hall

## References

  * [1] B. Feuerwehr, “Großbrand in Störfallbetrieb in Berlin-Marienfelde,” Berlin, Germany, 11.02.2021. [Online]. Available: https://www.berliner-feuerwehr.de/aktuelles/einsaetze/grossbrand-in-stoerfallbetrieb-in-berlin-marienfelde-3719
  * [2] T. Linder, V. Tretyakov, S. Blumenthal, P. Molitor, D. Holz, R. Murphy, S. Tadokoro, and H. Surmann, “Rescue robots at the collapse of the municipal archive of cologne city: A field report,” in _2010 IEEE Safety Security and Rescue Robotics_ , July 2010, pp. 1–6.
  * [3] F. Matsuno, N. Sato, K. Kon, H. Igarashi, T. Kimura, and R. Murphy, “Utilization of robot systems in disaster sites of the great eastern japan earthquake,” vol. 92, pp. 1–17, 12 2014.
  * [4] I. Kruijff-Korbayová, L. Freda, M. Gianni, V. Ntouskos, V. Hlaváč, V. Kubelka, E. Zimmermann, H. Surmann, K. Dulic, W. Rottner, and E. Gissi, “Deployment of ground and aerial robots in earthquake-struck amatrice in italy (brief report),” in _2016 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR)_ , Oct 2016, pp. 278–279.
  * [5] G.-J. M. Kruijff, F. Pirri, M. Gianni, P. Papadakis, M. Pizzoli, A. Sinha, V. Tretyakov, T. Linder, E. Pianese, S. Corrao, F. Priori, S. Febrini, and S. Angeletti, “Rescue robots at earthquake-hit mirandola, italy: A field report,” in _2012 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR)_ , Nov 2012, pp. 1–8.
  * [6] G. Kruijff, I. Kruijff-Korbayová, S. Keshavdas, B. Larochelle, M. Janíček, F. Colas, M. Liu, F. Pomerleau, R. Siegwart, M. Neerincx, R. Looije, N. Smets, T. Mioch, J. van Diggelen, F. Pirri, M. Gianni, F. Ferri, M. Menna, R. Worst, T. Linder, V. Tretyakov, H. Surmann, T. Svoboda, M. Reinštein, K. Zimmermann, T. Petříček, and V. Hlaváč, “Designing, developing, and deploying systems to support human-robot teams in disaster response,” _Advanced Robotics_ , vol. 28, no. 23, pp. 1547–1570, 2014.
  * [7] G. Steinbauer, J. Maurer, and A. Ciossek, “Field report: Civil protection exercise gas storage,” in _2014 IEEE International Symposium on Safety, Security, and Rescue Robotics (2014)_ , Oct 2014, pp. 1–2.
  * [8] N. Sato, “Evaluation exercise in the seminar for standard test method of ground robot at fukushima robot test field,” in _2021 IEEE International Conference on Intelligence and Safety for Robotics (ISR)_ , March 2021, pp. 127–130.
  * [9] S. Mayer, L. Lischke, and P. W. Woźniak, “Drones for Search and Rescue,” in _1st International Workshop on Human-Drone Interaction_. Glasgow, United Kingdom: Ecole Nationale de l’Aviation Civile [ENAC], May 2019. [Online]. Available: https://hal.archives-ouvertes.fr/hal-02128385
  * [10] H. Surmann, R. Worst, T. Buschmann, A. Leinweber, A. Schmitz, G. Senkowski, and N. Goddemeiner, “Integration of uavs in urban search and rescue missions,” in _2019 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR)_. IEEE, 2019, pp. 203–209.
  * [11] A. Gawel, R. Dubé, H. Surmann, J. I. Nieto, R. Siegwart, and C. Cadena, “3d registration of aerial and ground robots for disaster response: An evaluation of features, descriptors, and transformation estimation,” _CoRR_ , vol. abs/1709.00587, 2017. [Online]. Available: http://arxiv.org/abs/1709.00587
  * [12] S. Sumikura, M. Shibuya, and K. Sakurada, “Openvslam,” _Proceedings of the 27th ACM International Conference on Multimedia_ , Oct 2019. [Online]. Available: http://dx.doi.org/10.1145/3343031.3350539
  * [13] R. Mur-Artal and J. D. Tardos, “Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras,” _IEEE Transactions on Robotics_ , vol. 33, no. 5, p. 1255–1262, Oct 2017. [Online]. Available: http://dx.doi.org/10.1109/TRO.2017.2705103
  * [14] R. Bansal, G. Raj, and T. Choudhury, “Blur image detection using laplacian operator and open-cv,” in _2016 International Conference System Modeling Advancement in Research Trends (SMART)_ , Nov 2016, pp. 63–67.
  * [15] R. A. Pagaduan., M. C. R. Aragon., and R. P. Medina., “iblurdetect: Image blur detection techniques assessment and evaluation study,” in _Proceedings of the International Conference on Culture Heritage, Education, Sustainable Tourism, and Innovation Technologies - CESIT,_ , INSTICC. SciTePress, 2021, pp. 286–291.
  * [16] T. Sieberth, R. Wackrow, and J. H. Chandler, “Automatic detection of blurred images in uav image sets,” _ISPRS Journal of Photogrammetry and Remote Sensing_ , vol. 122, pp. 1–16, 2016. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0924271616303999
  * [17] B. T. Koik and H. Ibrahim, “A literature survey on blur detection algorithms for digital imaging,” in _2013 1st International Conference on Artificial Intelligence, Modelling and Simulation_ , 2013, pp. 272–277.
  * [18] O. Ozyesil, V. Voroninski, R. Basri, and A. Singer, “A survey of structure from motion,” 2017.
  * [19] Y. Furukawa and C. Hernández, “Multi-view stereo: A tutorial,” _Foundations and Trends® in Computer Graphics and Vision_ , vol. 9, no. 1-2, pp. 1–148, 2015. [Online]. Available: http://dx.doi.org/10.1561/0600000052

Figure 12: 360 degree panorama images of the 15 positions (2 - 16).