# Extension of Theory of Gravitomagnetism and Spinor Quantum Mechanics with
Dynamics of Free Electromagnetic Field

Dalibor Javůrek<EMAIL_ADDRESS>
###### Abstract

The theory of Gravitomagnetism and spinor quantum mechanics describing the
interaction between the Dirac spinor field, the electromagnetic field, and a
weak gravitational field is extended by including the Lagrangian density of
the free electromagnetic field. It is shown that the newly added term in the
Lagrangian density is necessary to restore a symmetric energy-momentum tensor
in the interaction term of the Lagrangian density within the theory. We prove
that when the electromagnetic energy term is missing, the tensor in the
interaction Lagrangian density does not represent the energy-momentum tensor
of the Dirac field minimally coupled to the electromagnetic field on flat
space-time. Moreover, it follows that the Lagrangian density remains unchanged
regardless of whether the theory is developed from the principle of general
covariance or is defined as a flat space-time theory with the gravitational
field treated as a regular tensor field, in analogy to the electromagnetic
field. This contradicts previous findings.

##  I Introduction

In the exploration of classical systems within external gravitational fields,
extensive attention has been devoted to a variety of gravitational effects.
Recent investigations have successfully affirmed the presence of the
gravitomagnetic, or frame-dragging, field induced by the Earth’s rotation.
This phenomenon was conclusively observed in two distinct experiments,
involving LAGEOS satellites [1, 2] and the Gravity Probe B (GP-B) satellite
[3, 4, 5, 6]. LAGEOS provided a measurement of the gravitomagnetic interaction
through its influence on satellite orbits, while the GP-B satellite validated
the general relativity prediction for the gravitomagnetic precession of a
gyroscope in Earth’s orbit. The LAGEOS effects originated from Lorentz-like
forces in the geodesic equation, whereas the GP-B effects were attributed to
Larmor-like torques from the spin precession equation. These distinctions
rendered the two experiments independent and mutually complementary [7].

In the realm of quantum systems interacting with gravitational fields, notable
experiments and theoretical studies have been conducted in relation to Earth’s
gravitational influence. These investigations include studies of neutrons
interacting with the Earth’s Newtonian field [8, 9] as well as atom
interferometer experiments aimed at rigorously testing the equivalence
principle [10]. Moreover, a gravitational wave detector composed of two
separated atom interferometers has been proposed [11]. Although attempts to
discern gravitomagnetic effects with these devices have been discussed, the
formidable challenges arising from the effects’ small scale and their
similarity to classical rotational effects make such experiments inherently
difficult [7].

The Dirac field in curved space-time represents a key area of research that
connects quantum mechanics, quantum field theory, and general relativity. Its
importance lies in its applications in understanding the universe,
particularly in astrophysics and cosmology. Additionally, it allows for the
discovery of novel types of effects following from the interaction of curved
space-time, the quantum-mechanical Dirac field, and the electromagnetic field.
Hopefully, the research in this area will indicate whether the wave-function
of a quantum system is much more closely related to the gravitational
interaction than previously expected [12, 13].

Recent research on Dirac fields in curved space-time has expanded our
knowledge in several ways. A rigorous while still pedagogically valuable study
has looked into how quantized Dirac fields behave in curved space-times with
low relativistic effects [14]. The influence of two-dimensional space
curvature on massless Dirac particles has been examined [15], and the criteria
for quantizing Dirac fields in curved space-times have been set [16].
Attention has been devoted to the exploration of quantum walks related to
massless Dirac fermions in curved space-time [17] and to entropic dynamics
[18]. Detailed analyses of solutions and oscillator models in curved
environments have been performed [19, 20, 21]. The research demonstrates the
ongoing effort to understand how quantum fields interact with the structure of
space-time.

Many other interesting effects resulting from the gravitational field coupling
to the Dirac field have been studied. Attention has been devoted to the
investigation of parity violation due to the Dirac field’s gravity-spin
interaction [22]. The interplay between Einstein’s equivalence principle,
atomic transition frequencies, and the g-factor measurements has been explored
[23]. The symmetry guaranteeing the same gravitational attraction for
electrons and positrons has been found [24]. Despite all the knowledge already
obtained about the topic, we believe there are still remarkable questions and
areas yet unexplored.

The paper analyzes and extends the theory of Gravitomagnetism and spinor
quantum mechanics (GQM) [7], which covers the interaction of the Dirac field,
the electro-magnetic field and the weak gravitational field. All fields in the
theory are assumed to be classical. We find the original GQM theory remarkable
and highly valuable. Its value lies in the development of the theory from
principles of general covariance and an effort to analyse, if the theory can
be transformed into a flat space-time theory. This could be highly beneficial
for physicists already familiar with flat space-time quantum field theory. In
the flat space-time theory, the gravitational field tensor describing
disturbance in the metric is treated as an ordinary tensor; in analogy to the
electro-magnetic field. It has been concluded [7], that when the theory is
developed from principles of general covariance, purely geometric terms
related to the metric appear in the total Lagrangian density. It follows, the
theory cannot be described by a Lagrangian density of a flat space-time theory
in a physically satisfactory way. In this paper, we show, that the Lagrangian
density of the Gravitomagnetic theory can be recast into the Lagrangian
density of a physically meaningful flat space-time theory if the original
theory is extended by a term related to free electro-magnetic field. We
consider this finding to be significant, since it may allow the theory to be
more accessible to physicists already acquainted with flat space-time
theories. Moreover, we show that the extension is necessary in order to obtain
symmetric energy-momentum tensor in the interaction term of the theory.

After a brief introduction to the topic has been given in Sec. I, we summarize
the GQM theory, as it has been developed by Adler et. al. in Sec. II. In Sec.
III together with Appendix A, we derive and analyze the energy-momentum tensor
of the corresponding flat space-time theory, which finds its place in the GQM
theory. This is essential for the physical interpretation of the Lagrangian
density of the theory and for the definition of the Gravitomagnetic theory as
a flat space-time theory. In Sec. IV, the Lagrangian density of the
Gravitomagnetic theory is extended with a term related to the free electro-
magnetic field. It is shown, that the addition of this term is necessary in
order to obtain a symmetric energy-momentum tensor in the interaction
Lagrangian density. In Sec. V, the equivalence between the definition of the
theory from the principle of general covariance and definition as a flat
space-time theory with the gravitation field treated as a tensor field is
shown. Sec. VI summarizes all findings mentioned in the paper. Appendix A is
devoted to the derivation of the flat space-time energy-momentum tensor
related to the non-extended Gravitomagnetic theory.

##  II Theory of Gravitomagnetism and Spinor Quantum Mechanics

A theory describing interaction between the electro-magnetic four-potential
field $A^{\mu}$ and the Dirac spinor field $\psi$ on weakly curved space-time
has been developed by Adler et. al. [7]. Since the gravitational field has
been assumed weak, coefficients of the metric tensor $g_{\mu\nu}$ are defined
by equation

$g_{\mu\nu}=\eta_{\mu\nu}+h_{\mu\nu}.$ (1)

$\eta_{\mu\nu}=\mathrm{diag}(1,-1,-1,-1)$ is the metric tensor of the flat
space-time and the coefficients of the tensor $h_{\mu\nu}$ describe small
perturbations on otherwise flat space-time. The coefficients $h_{\mu\nu}$ are
assumed to be much smaller than one $|h_{\mu\nu}|\ll 1$ in absolute value. The
matrix of the metric disturbance tensor $h_{\mu\nu}$ is given by the equation

$h_{\mu\nu}=\left[\begin{array}[]{cccc}2\phi&h^{1}&h^{2}&h^{3}\\\
h^{1}&2\phi&0&0\\\ h^{2}&0&2\phi&0\\\ h^{3}&0&0&2\phi\end{array}\right],$ (2)

where $\phi$ denotes the scalar gravitational potential and
$\mathbf{h}\leavevmode\nobreak\ =\leavevmode\nobreak\ [h^{1},h^{2},h^{3}]$ is
the gravitational vector potential. Geometrized units
$c=\hbar=\mu_{0}=\varepsilon_{0}=1$ are used in the paper. We remark, that
this choice of $h_{\mu\nu}$ tensor results in the geodesic equation (for small
speeds and terms linear in $h_{\mu\nu}$)

$\dfrac{d^{2}x^{i}}{dt^{2}}=a^{i}\approx-\Gamma^{i}_{00}=\partial_{t}h^{i}-\partial_{i}\phi.$
(3)

We further note, that the theory assumes, that the magnetic field changes
slowly with time , thus $\partial_{t}h^{i}=0$.

The theory [7] has been developed with the principle of general covariance.
Particularly, the GQM theory has been based upon the generalization of the
Lagrangian density of the Dirac field $\psi$ minimally coupled to the electro-
magnetic field $A^{\mu}$ on flat space-time to a general space-time by the
condition, that the gravitational field associated with the curved space-time
is weak. It has also been assumed, that the sources of the gravitational field
are moving slowly. After simplification and rearranging of terms in the
Lagrangian density of the theory $L$, it has been shown that the Lagrangian
density $L$ attains a form

$\displaystyle L$ $\displaystyle=L_{\rm flat}+L_{\rm IG}$ (4) $\displaystyle
L_{\rm flat}$
$\displaystyle=\dfrac{i}{2}[\bar{\psi}\gamma^{\mu}(\partial_{\mu}\psi)-(\partial_{\mu}\bar{\psi})\gamma^{\mu}\psi]$
$\displaystyle-m\bar{\psi}\psi-qA_{\mu}\bar{\psi}\gamma^{\mu}\psi$ (5)
$\displaystyle L_{\rm IG}$
$\displaystyle=-\dfrac{h_{\mu\nu}}{4}\left[\bar{\psi}\hat{\gamma}^{\nu}(i\partial^{\mu}\psi-
qA^{\mu}\psi)-(i\partial^{\mu}\bar{\psi}+qA^{\mu}\bar{\psi})\hat{\gamma}^{\nu}\psi\right]$
$\displaystyle=-\dfrac{1}{2}h_{\mu\nu}T^{\mu\nu};$ (6) $\displaystyle
T^{\mu\nu}$
$\displaystyle=\dfrac{i}{4}(\bar{\psi}\gamma^{\mu}\partial^{\nu}\psi-\partial^{\nu}\bar{\psi}\gamma^{\mu}\psi+\bar{\psi}\gamma^{\nu}\partial^{\mu}\psi-\partial^{\mu}\bar{\psi}\gamma^{\nu}\psi)+$
$\displaystyle-\dfrac{q}{2}(\bar{\psi}A^{\nu}\gamma^{\mu}\psi+\bar{\psi}A^{\mu}\gamma^{\nu}\psi).$
(7)

The $L_{\rm flat}$ is the Lagrangian density of the Dirac field $\psi$
minimally coupled to the electro-magnetic field $A^{\mu}$ on flat space-time,
$L_{\rm IG}$ is the interaction Lagrangian density describing the interaction
of the gravitational field $h_{\mu\nu}$ with the electro-magnetic field
$A^{\mu}$ and the Dirac field $\psi$; $\gamma^{\mu}$ are flat space-time Dirac
matrices obeying the anti-commutation relation

$[\gamma^{\mu},\gamma^{\mu}]_{+}=\gamma^{\mu}\gamma^{\nu}+\gamma^{\nu}\gamma^{\mu}=2\eta^{\mu\nu}.$
(8)

The field $\bar{\psi}$ is related to the Dirac field $\psi$ through the
relation $\bar{\psi}=\psi^{\dagger}\gamma^{0}$. It has been shown [7], that
the interaction Lagrangian density $L_{\rm IG}$ can be written as a scalar
product between the gravitational field tensor $h_{\mu\nu}$ and the tensor
$T^{\mu\nu}$; in the same manner as the interaction Lagrangian between the
electro-magnetic field $A^{\mu}$ and the electric four-current
$J^{\mu}=q\bar{\psi}\gamma^{\mu}\psi$. The tensor $T^{\mu\nu}$ has been
claimed to be the energy-momentum tensor of the Dirac field minimally coupled
to the electro-magnetic field $A^{\mu}$ on flat space-time [7].

The action of the theory $S$ is given by a volume integral over the whole
space-time, which is considered to be weakly curved

$S=\int d^{4}x\sqrt{-g}\,L.$ (9)

In Eq. (9) $g$ denotes the determinant of the metric tensor $g_{\mu\nu}$. In
accordance with Adler et. al., let us denote the overall Lagrangian density
under the integral in Eq. (9) as

$\mathcal{L}=\sqrt{-g}\,L.$ (10)

Physically, the quantity $\mathcal{L}$ can be considered the Lagrangian
density of a theory on flat space-time $x^{\mu}$. However, it has to be kept
in mind that the coordinates $x^{\mu}$ are not physical.

##  III Energy-Momentum Tensor of the Non-Extended Theory

Knowledge of the energy-momentum tensor of a theory describing the Dirac field
minimally coupled to the electro-magnetic field on flat space-time is
essential, because of its identification in the Lagrangian density of the GQM
theory $L_{\rm IG}$, see Eq. (6). Moreover, the tensor appearing in the
interaction Lagrangian density of the GQM theory is required to be a symmetric
energy-momentum tensor, since it could be considered as a source of the tensor
gravitational field $h_{\mu\nu}$ 111if the Lagrangian density of the free weak
gravitational field $h_{\mu\nu}$ were included into the overall Lagrangian
density $L$, in accordance with the Einstein equations of the general theory
of relativity. In this section, we prove that the energy-momentum tensor
$T^{\mu\nu}$ of the GQM theory, see Eq. (7), does not satisfy the required
relation with the canonical energy-momentum $\Theta^{\mu\nu}$. Therefore, it
is not the energy-momentum tensor of the corresponding flat space-time theory.
We show, that the energy-momentum tensor derived with the second Noether
theorem satisfying this relation is distinct and asymmetric. We comment on the
asymmetric nature of the tensor.

In paper [7], where the GQM theory has been developed, the authors derived
energy-momentum tensor $T^{\mu\nu}$ of the Dirac field minimally coupled to
the electro-magnetic field $A^{\mu}$, see appendix B in [7]. They started with
the derivation of the canonical energy-momentum tensor for the free Dirac
field on flat space-time. Then, they correctly stated, that the canonical
energy-momentum tensor can be symmetrized into the form in Eq. (B3) in [7]. To
include the electro-magnetic interaction into the energy-momentum tensor, they
exchanged all the derivatives multiplied by an imaginary unit
$i\partial_{\mu}$ in the tensor for $i\partial_{\mu}-qA_{\mu}$. They arrived
at a symmetric energy-momentum tensor $T^{\mu\nu}$ in Eq. (7), which can be
rewritten into the form

$\displaystyle T^{\mu\nu}$
$\displaystyle=\dfrac{1}{2}\left(\Theta^{\mu\nu}+\Theta^{\nu\mu}-qA^{\mu}\bar{\psi}\gamma^{\nu}\psi-
qA^{\nu}\bar{\psi}\gamma^{\mu}\psi\right);$ (11)
$\displaystyle\Theta^{\mu\nu}$
$\displaystyle=\dfrac{i}{2}\left[\bar{\psi}\gamma^{\mu}(\partial^{\nu}\psi)-(\partial^{\nu}\bar{\psi})\gamma^{\mu}\psi\right],$
(12)

where $\Theta^{\mu\nu}$ denotes the canonical energy-momentum tensor of the
free Dirac field, which is not symmetric. The validity of the energy-momentum
tensor in Eq. (11) was justified by the calculation of its divergence

$\partial_{\mu}T^{\mu\nu}=-q(\bar{\psi}\gamma_{\mu}\psi)F^{\nu\mu}=-F^{\nu\mu}J_{\mu}.$
(13)

where $F^{\nu\mu}$ is the electro-magnetic field tensor on flat space-time.

We derive the energy-momentum tensor $T^{\mu\nu}_{\rm N}$ from the Lagrangian
density $L_{\rm flat}$ in Eq. (5) with the second Noether theorem [26]. We
utilize the theorem, since it is considered to provide symmetric energy-
momentum tensors [26]. The derived energy-momentum tensor $T^{\mu\nu}_{\rm N}$
is given by the equation

$T^{\mu\nu}_{\rm
N}=\dfrac{1}{2}(\Theta^{\mu\nu}+\Theta^{\nu\mu}+q\bar{\psi}A^{\nu}\gamma^{\mu}\psi-q\bar{\psi}A^{\mu}\gamma^{\nu}\psi).$
(14)

For details about the derivation of the tensor, please, see Appendix A. The
tensor $T^{\mu\nu}_{\rm N}$ in Eq. (14) consists of a symmetric part
$\Theta^{\mu\nu}+\Theta^{\nu\mu}$ and a remaining anti-symmetric part,
explicitly dependent on components of the electro-magnetic field $A^{\mu}$.
Therefore, the obtained overall energy-momentum tensor $T^{\mu\nu}_{\rm N}$ is
not symmetric and not equal to the energy-momentum tensor $T^{\mu\nu}$
obtained in Eq. (11). The asymmetry in the tensor $T^{\mu\nu}_{\rm N}$ occurs,
because the electro-magnetic field $A^{\mu}$ transfers its angular momentum to
the Dirac field, but its intrinsic angular momentum is not accounted for in
the tensor $T^{\mu\nu}_{\rm N}$. This occurs, since the Lagrangian density
$L_{\rm flat}$ in Eq. (5) lacks a term related to the free electro-magnetic
field. In this work, we define the electro-magnetic field tensor $F^{\mu\nu}$
on flat space-time by the equation

$F^{\mu\nu}=\partial^{\mu}A^{\nu}-\partial^{\nu}A^{\mu}.$ (15)

There are two distinct energy momentum tensors calculated for the theory
defined by the Lagrangian density $L_{\rm flat}$ in Eq. (5) . The symmetric
one $T^{\mu\nu}$ in Eq. (11), derived by the authors in [7], and an asymmetric
one $T^{\mu\nu}_{\rm N}$ in Eq. (14). We prove, that the energy-momentum
tensor $T^{\mu\nu}$ in Eq. (11) is not the energy-momentum tensor related to
the flat space-time theory described by the Lagrangian density $L_{\rm flat}$
in Eq. (5). This follows from the expression for the four-divergence of the
tensor $T^{\mu\nu}$ in Eq. (13). It has to be equal to four-divergence of
canonical energy-momentum tensor $\Theta^{\mu\nu}$ (see Eq. (47)), which it is
not

$\partial_{\mu}T^{\mu\nu}\neq\partial_{\mu}\Theta^{\mu\nu}=-\dfrac{\partial
L_{\rm flat}}{\partial A_{\mu}}(\partial^{\nu}A_{\mu}).$ (16)

For details, please, see Appendix A. On the other hand, the four-divergence of
the energy-momentum tensor $T^{\mu\nu}_{\rm N}$ satisfies the first equality
in Eq. (16).

In consequence, the tensor $T^{\mu\nu}$ in the interaction Lagrangian $L_{\rm
IG}$ in Eq. (6) is not the energy-momentum tensor related to the flat space-
time theory. On the other hand, the tensor $T^{\mu\nu}_{\rm N}$ is the energy-
momentum tensor of the theory, since its four-divergence is equal to the four-
divergence of the canonical energy-momentum tensor $\Theta^{\mu\nu}$. As an
energy-momentum tensor, it can be written as an extension of the canonical
momentum tensor $\Theta^{\mu\nu}$ with the four-divergence of the Belinfante
tensor $B^{\lambda\mu\nu}$, see Eq. (36). Thus, the asymmetric tensor
$T^{\mu\nu}_{\rm N}$ is the energy-momentum tensor related to the flat space-
time theory defined by Lagrangian density $L_{\rm flat}$ in Eq. (5). A
symmetric energy-momentum tensor in the interaction term of the Lagrangian
density $L_{\rm IG}$ appears if the theory is extended with the dynamics of
the free electro-magnetic field.

##  IV Extension of the Theory with Lagrangian Density of Free Electromagnetic
Field

To obtain a symmetric energy-momentum tensor $T^{\mu\nu}_{\rm N}$ with the
second Noether theorem from a flat space-time theory based on the Lagrangian
density $L_{\rm flat}$ in Eq. (5), it is necessary to extend the Lagrangian
density with Lagrangian density of free electro-magnetic field $A^{\mu}$.
Then, we obtain Lagrangian density of quantum electrodynamics on flat space-
time [26]

$L_{\rm QED,flat}=L_{\rm flat}-\dfrac{1}{4}F^{\mu\nu}F_{\mu\nu}.$ (17)

The energy-momentum tensor derived from the Lagrangian density in Eq. (17)
with the second Noether theorem is equal to [26]

$\displaystyle T^{\mu\nu}_{\rm QED}$ $\displaystyle=T^{\mu\nu}_{\rm
N}+F^{\mu}_{\phantom{\mu}\lambda}F^{\lambda\nu}+\dfrac{\eta^{\mu\nu}}{4}F^{\rho\lambda}F_{\rho\lambda}-qA^{\nu}\bar{\psi}\gamma^{\mu}\psi$
$\displaystyle=\Theta^{\mu\nu}+\Theta^{\nu\mu}-\dfrac{q}{2}(A^{\nu}\bar{\psi}\gamma^{\mu}\psi+A^{\mu}\bar{\psi}\gamma^{\nu}\psi)+F^{\mu}_{\phantom{\mu}\lambda}F^{\lambda\nu}$
$\displaystyle-\eta^{\mu\nu}L_{\rm QED,flat}.$ (18)

We note, that $L_{\rm flat}\equiv 0$ due to the Euler-Lagrange equations
(Dirac equations) for $\psi$ and $\bar{\psi}$. Therefore in Eq. (18), we may
put the Lagrangian density of quantum electrodynamics $L_{\rm QED,flat}$ equal
to the Lagrangian density of the free electro-magnetic field

$L_{\rm QED,flat}=-\dfrac{1}{4}F^{\mu\nu}F_{\mu\nu}.$ (19)

The tensor $T^{\mu\nu}$ in Eq. (11) of the GQM theory, obtained in the
interaction Lagrangian density $L_{\rm IG}$ in Eq. (6), constitutes a part of
the energy momentum tensor $T^{\mu\nu}_{\rm QED}$ in Eq. (18). Particularly,

$T^{\mu\nu}_{\rm
QED}=T^{\mu\nu}+F^{\mu}_{\phantom{\mu}\lambda}F^{\lambda\nu}-\eta^{\mu\nu}L_{\rm
QED,flat}.$ (20)

Therefore, it is tempting to extend the Lagrangian density of the GQM theory
$L$ in Eq. (4) with the Lagrangian density of the free electro-magnetic field
on weakly curved space-time

$L_{\rm
EM}=-\dfrac{1}{4}F^{\mu\nu}F_{\mu\nu}=-\dfrac{1}{4}g^{\mu\lambda}g^{\nu\rho}F_{\lambda\rho}F_{\mu\nu}$
(21)

and see, if the energy-momentum tensor $T^{\mu\nu}_{\rm QED}$ emerges in the
interaction Lagrangian density $L_{\rm IG}$ instead of the tensor
$T^{\mu\nu}$.

The contra-variant metric tensor $g^{\mu\nu}$ has approximately the form

$g^{\mu\nu}\approx\eta^{\mu\nu}-\eta^{\mu\rho}\eta^{\nu\lambda}h_{\rho\lambda}\approx\eta^{\mu\nu}-h^{\mu\nu}.$
(22)

In the approximation, the second and higher order terms in the gravitational
field tensor components $h_{\mu\nu}$ are neglected. The tensor of the electro-
magnetic field $F_{\mu\nu}$ on a curved space-time is defined with covariant
derivatives $\nabla_{\mu}$ by the equation [27]

$F_{\mu\nu}=\nabla_{\mu}A_{\nu}-\nabla_{\nu}A_{\mu}.$ (23)

But due to the symmetry of the Christoffel symbols
$\Gamma^{\nu}_{\mu\rho}\leavevmode\nobreak\ =\leavevmode\nobreak\
\Gamma^{\nu}_{\rho\mu}$ [27], the electro-magnetic field tensor $F_{\mu\nu}$
attains the same form as on flat-space-time

$F_{\mu\nu}=\nabla_{\mu}A_{\nu}-\nabla_{\nu}A_{\mu}=\partial_{\mu}A_{\nu}-\partial_{\nu}A_{\mu}.$
(24)

If we substitute for the metric tensor $g^{\mu\nu}$ from Eq. (22) into Eq.
(21) and use the anti-symmetric property of the electro-magnetic field tensor
$F_{\mu\nu}=-F_{\nu\mu}$ and the symmetric property of the gravitational field
tensor $h_{\mu\nu}=h_{\nu\mu}$, we obtain the Lagrangian density of the
electro-magnetic field $L_{\rm EM}$ up to the first order in $h_{\mu\nu}$

$L_{\rm
EM}\approx-\dfrac{1}{4}\eta^{\mu\rho}\eta^{\nu\lambda}F_{\rho\lambda}F_{\mu\nu}-\dfrac{1}{2}F^{\mu\lambda}F_{\lambda}^{\phantom{\lambda}\nu}h_{\mu\nu}.$
(25)

The first term is equal to the Lagrangian density of the electro-magnetic
field on flat space-time, due to Eq. (24). The second term in Eq. (25)
describes the interaction of the electro-magnetic field $A^{\mu}$ with the
gravitational tensor field $h_{\mu\nu}$.

Since the Lagrangian density $L_{\rm EM}$ is added to Lagrangian density of
the GQM theory $L$ (see Eq. (4)), we transfer the second term in Eq. (25) into
the interaction Lagrangian density $L_{\rm IG}$, see Eq. (6), where it
reconstructs part of the energy-momentum tensor $T^{\mu\nu}_{\rm QED}$ in Eq.
(20) related to the free electro-magnetic field. As a result, we can write the
Lagrangian density of quantum electrodynamics on a weakly curved space-time
$L$ as the sum of the Lagrangian density of the same theory on a flat space-
time and an interaction term related to the tensor gravitational field
$h_{\mu\nu}$

$L=L_{\rm
QED,flat}-\dfrac{1}{2}h_{\mu\nu}(T^{\mu\nu}+F^{\mu\lambda}F_{\lambda}^{\phantom{\lambda}\nu}).$
(26)

The tensor, which multiplies the gravitational field tensor $h_{\mu\nu}$ in
the second term of Eq. (26) is missing the diagonal term $-\eta^{\mu\nu}L_{\rm
QED,flat}$ in order to fully reconstruct the energy-momentum tensor of quantum
electrodynamics $T^{\mu\nu}_{\rm QED}$ in Eq. (20).

In order to rectify this situation, it is necessary to work with the
Lagrangian density

$\mathcal{L}=L\sqrt{-g},$ (27)

which appears under the four-dimensional volume integral for the action $S$ in
Eq. (9). The Lagrangian density $\mathcal{L}$ can be considered as defining a
flat space-time theory on coordinates $x^{\mu}$, which are not physical.

##  V Equivalence of Gravitomagnetic Theory with Flat Space-Time Theory

It has been noted [7], that the GQM theory posses purely geometric term in its
total Lagrangian density $\mathcal{L}$ in Eq. (27) when developed from
principles of general covariance. Thus, if the weak gravitational field
$h_{\mu\nu}$ was treated as a tensor field propagating on a flat space-time
coupled to other fields on flat space-time in the definition of this theory,
some terms in the resulting Euler-Lagrange equation for the Dirac field $\psi$
would be missing. On the other hand, Feynman, Weinberg and Schwinger have all
been pointing out, that the geometric nature of the gravitational field does
not have to be taken into account [7, 28, 29] and can be treated as an
ordinary field on flat space-time as far as the gravitational field is weak.
Here, we show the equivalence between the GQM theory extended with a free
electro-magnetic term (see SEC III.) and an idea proposed by Feynman, Weinberg
and Schwinger.

We start with calculation of term $\sqrt{-g}$. Up to the first order in
components of gravitational field tensor $h_{\mu\nu}$. It is equal to [7]

$\sqrt{-g}\approx 1+\dfrac{1}{2}h^{\mu}_{\phantom{\mu}\mu}\approx
1+\dfrac{1}{2}\eta^{\mu\nu}h_{\mu\nu}$ (28)

The Lagrangian density $\mathcal{L}$ in Eq. (27) can be written up to the
first order in terms of gravitational field tensor $h_{\mu\nu}$ as

$\displaystyle\mathcal{L}\approx L_{\rm QED,flat}\sqrt{-g}+L_{\rm IG}.$ (29)

Here, we consider, that the theory includes the Lagrangian density of the
electro-magnetic field on weakly curved space-time. Therefore, we can rewrite
the Eq. (29) using the Eqs. (28) and (26) up to the first order in terms of
the gravitational field tensor $h_{\mu\nu}$ as

$\displaystyle\mathcal{L}$ $\displaystyle\approx L_{\rm
QED,flat}-\dfrac{1}{2}h_{\mu\nu}(T^{\mu\nu}+F^{\mu\lambda}F_{\lambda}^{\phantom{\lambda}\nu}-\eta^{\mu\nu}L_{\rm
QED,flat})$ (30) $\displaystyle\approx L_{\rm
QED,flat}-\dfrac{1}{2}h_{\mu\nu}T^{\mu\nu}_{\rm QED}.$ (31)

In Eq. (30), we have identified, that the tensor in the product with
gravitational field tensor $h_{\mu\nu}$ is the energy-momentum tensor of
quantum electrodynamics on flat space-time $T^{\mu\nu}_{\rm QED}$. Therefore,
the gravitational field $h_{\mu\nu}$ has a source $T^{\mu\nu}_{\rm QED}$ in
analogy with the electro-magnetic radiation’s conserved four-current $J^{\mu}$
in flat space-time.

The Lagrangian density $\mathcal{L}$ in Eq. (31) can be interpreted to
describe a theory on a flat space-time $x^{\mu}$ involving the tensor
gravitational field $h_{\mu\nu}$, the electro-magnetic field $A^{\mu}$ and the
Dirac fields $\psi$ and $\bar{\psi}$. We have shown, that the GQM theory
proposed by Adler et. al. extended with the dynamics of the free electro-
magnetic field is equivalent to a physically meaningful flat space-time
theory. Nevertheless, the flat space-time described by coordinates $x^{\mu}$
is strictly speaking not physical, but it is related to the physical one by a
non-trivial metric tensor $g_{\mu\nu}$. At last it is worth to note, that the
addition of the free electro-magnetic term to the Lagrangian density $L$
allows the theory to predict propagation of the electro-magnetic field on
weakly curved space-time.

##  VI Conclusion

The energy-momentum tensor of the original GQM theory presented in Sec. II has
been shown to be asymmetric due to the lack of a term describing the intrinsic
energy of the electro-magnetic field in the Lagrangian density. Thus, the
energy-momentum tensor does not appear in the interaction Lagrangian density
of the original GQM theory. After the extension of the Lagrangian density of
the GQM theory with the free electro-magnetic field term on a weakly curved
space-time, a symmetric energy-momentum tensor of flat space-time quantum
electrodynamics appeared in the interaction Lagrangian density. It has been
shown, that the Lagrangian density of the theory developed from the principle
of general covariance [7] extended by the free electro-magnetic field term is
equivalent to a flat space-time theory, where the gravitational field is
treated as an ordinary tensor field, in analogy to the electro-magnetic field.

##  Appendix A Derivation of Energy Momentum Tensor $T^{\mu\nu}_{\rm N}$ with
Second Noether Theorem

First we introduce the canonical energy-momentum tensor $\Theta^{\mu\nu}$.
Then, we show which requirements are put on a general energy-momentum tensor
$T^{\mu\nu}$, which can replace the canonical energy-momentum tensor
$\Theta^{\mu\nu}$ in a studied theory. Finally, we derive the energy-momentum
tensor $T^{\mu\nu}_{\rm N}$ from the Lagrangian density $L_{\rm QED,flat}$ in
Eq. (17) with the second Noether theorem and discuss, wether it still holds
properties put on a generalized energy-momentum tensor.

The canonical energy-momentum tensor $\Theta^{\mu\nu}$ can be derived from the
Lagrangian density $L$ in Eq. (4) as follows

$\displaystyle\dfrac{\partial L}{\partial x^{\nu}}$
$\displaystyle=\dfrac{\partial L}{\partial\psi}\dfrac{\partial\psi}{\partial
x^{\nu}}+\dfrac{\partial
L}{\partial(\partial_{\mu}\psi)}\dfrac{\partial^{2}\psi}{\partial
x^{\mu}\partial x^{\nu}}+$ $\displaystyle\dfrac{\partial
L}{\partial\bar{\psi}}\dfrac{\partial\bar{\psi}}{\partial
x^{\nu}}+\dfrac{\partial
L}{\partial(\partial_{\mu}\bar{\psi})}\dfrac{\partial^{2}\bar{\psi}}{\partial
x^{\mu}\partial x^{\nu}}+$ $\displaystyle\dfrac{\partial{L}}{\partial
A_{\mu}}\dfrac{\partial A_{\mu}}{\partial x^{\nu}}.$ (32)

With the utilization of the Euler-Lagrange equations for the Dirac field
$\psi$

$\dfrac{\partial L}{\partial\psi}-\dfrac{\partial}{\partial
x^{\mu}}\left(\dfrac{\partial L}{\partial(\partial_{\mu}\psi)}\right)=0$ (33)

and the same equation, but for the field $\bar{\psi}$, we arrive at the
equation

$\displaystyle\dfrac{\partial L}{\partial x^{\nu}}=\dfrac{\partial}{\partial
x^{\mu}}\left(\dfrac{\partial
L}{\partial(\partial_{\mu}\psi)}(\partial_{\nu}\psi)+\dfrac{\partial
L}{\partial(\partial_{\mu}\bar{\psi})}(\partial_{\nu}\bar{\psi})\right)+\dfrac{\partial{L}}{\partial
A_{\mu}}\dfrac{\partial A_{\mu}}{\partial x^{\nu}}.$ (34)

Equation (34) can be finally arranged into form, in which the canonical
energy-momentum tensor appears

$\displaystyle\dfrac{\partial}{\partial
x^{\mu}}\underbrace{\left(\dfrac{\partial
L}{\partial(\partial_{\mu}\psi)}(\partial_{\nu}\psi)+\dfrac{\partial
L}{\partial(\partial_{\mu}\bar{\psi})}(\partial_{\nu}\bar{\psi})-\delta^{\mu}_{\nu}L\right)}_{\Theta^{\mu}_{\phantom{\mu}\nu}}=$
$\displaystyle-\dfrac{\partial{L}}{\partial A_{\mu}}\dfrac{\partial
A_{\mu}}{\partial x^{\nu}}.$ (35)

We have assumed, that except the fields $\psi$ and $\bar{\psi}$, the
Lagrangian density $L$ contains an external electro-magnetic field $A^{\mu}$.
The electro-magnetic field is defined as a function of space and time and may
exert a force influencing the motion of the Dirac fields $\psi$ and
$\bar{\psi}$. The force term occupies the right-hand side of Eq. (35).
Therefore, in the studied case the canonical energy-momentum tensor
$\Theta^{\mu\nu}$ is not conserved, but its components are influenced by the
external force field.

In the general theory of relativity, only symmetric energy-momentum tensors
have physical significance. They act as a source of space-time curvature.
Therefore, it is desirable to symmetrize the (generally asymmetric) canonical
energy-momentum tensor $\Theta^{\mu\nu}$. The canonical energy-momentum tensor
$\Theta^{\mu\nu}$ defined by Eq. (35) and Lagrangian $L_{\rm flat}$ in Eq. (5)
is not symmetric. This occurs, because the fields $\psi$ and $\bar{\psi}$
possess intrinsic angular momentum – spin. The symmetric energy-momentum
tensors are the Hilbert and the Noether energy-momentum tensor derived by her
second theorem [30, 26]. In this paper, we derive the energy momentum tensor
by the second Noether theorem.

Energy-momentum tensor of a theory related to a particular Lagrangian density
is not given uniquely. The canonical energy momentum tensor $\Theta^{\mu\nu}$
of a theory can be extended by the four-divergence of the Belinfante tensor
$B^{\lambda\mu\nu}$ in order to obtain new energy momentum tensor $T^{\mu\nu}$

$T^{\mu\nu}=\Theta^{\mu\nu}+\partial_{\lambda}B^{\lambda\mu\nu}.$ (36)

It is required, that the divergence of the new energy-momentum tensor and the
canonical energy-momentum tensor remain the same

$\partial_{\mu}T^{\mu\nu}=\partial_{\mu}\Theta^{\mu\nu}.$ (37)

This holds if the Belinfante tensor $B^{\lambda\mu\nu}$ is anti-symmetric in
its first two indices

$B^{\lambda\mu\nu}=-B^{\mu\lambda\nu}.$ (38)

As a consequence of the anti-symmetry of the Belinfante tensor in Eq. (38),
the spatial-volume integrals of $T^{0\nu}$ and $\Theta^{0\nu}$ are equal

$\displaystyle P^{\nu}$ $\displaystyle=\int d^{3}x\,T^{0\nu}=$ (39)
$\displaystyle=\int d^{3}x\,\Theta^{0\nu}+\int
d^{3}x\,\partial_{0}B^{00\nu}+\int d^{3}x\,\partial_{j}B^{j0\nu}$ (40)
$\displaystyle=\int d^{3}x\,\Theta^{0\nu}.$ (41)

The second term in Eq. (40) is equal to zero since $B^{00\nu}=0$ due to the
antisymmetry of the Belinfante tensor in its first two indices. The third term
is equal to zero, since the volume of the integration is assumed to expand its
boundary to infinity and the terms $B^{j0\nu}$ are assumed to decrease faster
than $1/r^{2}$. The equality of volume integrals in Eqs. (39) and (41) is
necessary, since both energy-momentum tensors $\Theta^{\mu\nu}$ and
$T^{\mu\nu}$ have to provide the same values of four-momentum components
(observables) $P^{\nu}$.

According to Freese [26] the second Noether theorem states, that the energy-
momentum tensor $T^{\mu}_{\phantom{\mu}\nu,\mathrm{N}}$ is defined by the
equation

$\displaystyle T^{\mu}_{\phantom{\mu}\nu,\mathrm{N}}$
$\displaystyle=\mathscr{D}^{\mu}_{\phantom{\mu}\nu}[\psi,\bar{\psi}]-\delta^{\mu}_{\nu}L;$
(42) $\displaystyle\mathscr{D}^{\mu}_{\phantom{\mu}\nu}[\psi,\bar{\psi}]$
$\displaystyle=-\dfrac{\partial}{\partial(\partial_{\mu}\xi^{\nu})}\left[\dfrac{\partial
L}{\partial\psi}\Delta\psi+\dfrac{\partial
L}{\partial(\partial_{\lambda}\psi)}\Delta(\partial_{\lambda}\psi)\right.$
$\displaystyle\left.\dfrac{\partial
L}{\partial\bar{\psi}}\Delta\bar{\psi}+\dfrac{\partial
L}{\partial(\partial_{\lambda}\psi)}\Delta(\partial_{\lambda}\psi)\right].$
(43)

$\xi^{\mu}$ is a component of the local translational vector, $\Delta\psi$ and
$\Delta(\partial_{\lambda}\psi)$ are total variations of the spinor field
$\psi$ and its derivative $\partial_{\lambda}\psi$. For formulas defining the
total variations, please see [26]. It is worth to note, that the quantity
$\mathscr{D}^{\mu}_{\phantom{\mu}\nu}$ is a linear operator with respect to
the Lagrangian density $L$. Thus, the calculations can be conveniently
partitioned. Particularly, the energy-momentum tensor
$T^{\mu}_{\phantom{\mu}\nu,\mathrm{N}}$ for the Lagrangian density of the free
Dirac field $\psi$ and $\bar{\psi}$ on the flat space-time can be calculated
first. Then, the energy-momentum tensor calculated from the interaction
Lagrangian density (incorporating fields $\psi$, $\bar{\psi}$ and $A^{\mu}$)
has to be added to the one previously obtained. The final form of the energy-
momentum tensor is [31]

$\displaystyle T^{\mu\nu}_{\rm N}$
$\displaystyle=\dfrac{1}{2}(\Theta^{\mu\nu}+\Theta^{\nu\mu}+q\bar{\psi}A^{\nu}\gamma^{\mu}\psi-q\bar{\psi}A^{\mu}\gamma^{\nu}\psi);$
$\displaystyle\Theta^{\mu\nu}$
$\displaystyle=\dfrac{i}{2}\left[\bar{\psi}\gamma^{\mu}(\partial^{\nu}\psi)-(\partial^{\nu}\bar{\psi})\gamma^{\mu}\psi\right].$
(44)

The obtained tensor $T^{\mu\nu}_{\rm N}$ is not symmetric. This is caused by
the angular momentum transferred to the field $\psi$ by the external electro-
magnetic field $A^{\mu}$.

During derivation of the energy-momentum tensor $T^{\mu\nu}_{\rm N}$, the
Belinfante tensor can be identified with the expression

$\displaystyle B^{\lambda\mu\nu}$
$\displaystyle=\dfrac{i}{4}\left[-\dfrac{\partial
L}{\partial(\partial_{\lambda}\psi)}\sigma^{\mu\nu}\psi+\bar{\psi}\sigma^{\mu\nu}\dfrac{\partial
L}{\partial(\partial_{\lambda}\bar{\psi})}\right]$ (45)
$\displaystyle=\dfrac{1}{8}\bar{\psi}[\gamma^{\lambda},\sigma^{\mu\nu}]_{+}\psi;\,\sigma^{\mu\nu}=\dfrac{i}{2}[\gamma^{\mu},\gamma^{\nu}].$
(46)

From Eq. (45) follows, that the Blinfante tensor $B^{\lambda\mu\nu}$ is not
influenced by an interaction Lagrangian density, which is independent of
derivatives of spinors $\partial_{\lambda}\psi$ and
$\partial_{\lambda}\bar{\psi}$. This is the case of the interaction Lagrangian
density for the electro-magnetic field, see Eq. (5). It can be easily
verified, that the Belinfante tensor in Eq. (46) satisfies the requirement in
Eq. (38). Therefore, the derived energy-momentum tensor $T^{\mu\nu}_{\rm N}$
in Eq. (42) is connected with the canonical energy-momentum tensor
$\Theta^{\mu\nu}$ through the relation in Eq. (36). Moreover, the components
of the four-momentum $P^{\nu}$ are the same, regardless of the utilized
energy-momentum tensor. See Eqs. (39) and (41).

The explicit value of the four-divergence of the canonical energy-momentum
tensor $\Theta^{\mu\nu}$ and the energy momentum tensor $T^{\mu\nu}_{\rm N}$
derived by the second Noether theorem is equal to

$\partial_{\mu}\Theta^{\mu\nu}=\partial_{\mu}T^{\mu\nu}_{\rm
N}=-\dfrac{\partial L}{\partial
A_{\mu}}(\partial^{\nu}A_{\mu})=q(\partial^{\nu}A_{\mu})\bar{\psi}\gamma^{\mu}\psi$
(47)

in accordance with Eq. (35).

## References

  * Ciufolini  _et al._ [1997] I. Ciufolini, F. Chieppa, D. Lucchesi, and F. Vespe, Test of lense - thirring orbital shift due to spin, Classical and Quantum Gravity 14, 2701 (1997).
  * Ciufolini and Wheeler [1995] I. Ciufolini and J. A. Wheeler,  _Gravitation and Inertia_ (Princeton University Press, New Jersey, 1995).
  * Adler [2006] R. J. Adler,  _The New Physics for the Twenty-First Century_ , edited by G. Fraser (Cambridge University Press, Cambridge U.K., 2006).
  * Will [1993] C. M. Will,  _Theory and Experiment in Gravitational Physics_ (Cambridge University Press, Cambridge U.K., 1993).
  * Will [2018] C. M. Will,  _Theory and Experiment in Gravitational Physics_, 2nd ed. (Cambridge University Press, 2018).
  * Everitt  _et al._ [2011] C. W. F. Everitt, D. B. DeBra, B. W. Parkinson, J. P. Turneaure, J. W. Conklin, M. I. Heifetz, G. M. Keiser, A. S. Silbergleit, T. Holmes, J. Kolodziejczak, M. Al-Meshari, J. C. Mester, B. Muhlfelder, V. G. Solomonik, K. Stahl, P. W. Worden, W. Bencze, S. Buchman, B. Clarke, A. Al-Jadaan, H. Al-Jibreen, J. Li, J. A. Lipa, J. M. Lockhart, B. Al-Suwaidan, M. Taber, and S. Wang, Gravity probe b: Final results of a space experiment to test general relativity, Phys. Rev. Lett. 106, 10.1103/PhysRevLett.106.221101 (2011).
  * Adler  _et al._ [2012] R. J. Adler, P. Chen, and E. Varani, Gravitomagnetism and spinor quantum mechanics, Phys. Rev. D 85, 025016 (2012).
  * Nesvizhevsky  _et al._ [2002] V. V. Nesvizhevsky, H. G. Börner, A. K. Petukhov, H. Abele, S. Baeßler, F. J. Rueß, T. Stöferle, A. Westphal, A. M. Gagarski, G. A. Petrov, and A. V. Strelkov, Quantum states of neutrons in the earth’s gravitational field, Nature 415, 297 (2002).
  * Tobar  _et al._ [2022] G. Tobar, S. Haine, F. Costa, and M. Zych, Mass-energy equivalence in gravitationally bound quantum states of the neutron, Physical Review A 106, 10.1103/physreva.106.052801 (2022).
  * Zych and Brukner [2015] M. Zych and C. Brukner, Quantum formulation of the einstein equivalence principle, (2015), arXiv:1502.00971 [gr-qc] .
  * Dimopoulos  _et al._ [2008] S. Dimopoulos, P. W. Graham, J. M. Hogan, M. A. Kasevich, and S. Rajendran, Atomic gravitational wave interferometric sensor, Phys. Rev. D 78, 122002 (2008).
  * Maldacena and Susskind [2013] J. Maldacena and L. Susskind, Cool horizons for entangled black holes, Fortschritte der Physik 61, 781 (2013).
  * Susskind [2016] L. Susskind, Copenhagen vs Everett, teleportation, and ER=EPR, Fortschritte der Physik 64, 551 (2016).
  * Falcone and Conti [2022] R. Falcone and C. Conti, Nonrelativistic limit of scalar and dirac fields in curved spacetime, Physical Review D 10.1103/PhysRevD.107.045012 (2022).
  * Flouris  _et al._ [2018] K. Flouris, M. M. Jimenez, J. Debus, and H. J. Herrmann, Confining massless dirac particles in two-dimensional curved space, Physical Review B 10.1103/PhysRevB.98.155419 (2018).
  * Cortez  _et al._ [2020] J. Cortez, B. E. Navascu’es, G. M. M. Marugán, S. Prado, and J. Velhinho, Uniqueness criteria for the fock quantization of dirac fields and applications in hybrid loop quantum cosmology, Universe 10.3390/universe6120241 (2020).
  * Molfetta  _et al._ [2013] G. D. Molfetta, M. Brachet, and F. Debbasch, Quantum walks as massless dirac fermions in curved space-time, Physical Review A 88, 042301 (2013).
  * Ipek  _et al._ [2019] S. Ipek, M. Abedi, and A. Caticha, Entropic dynamics: reconstructing quantum field theory in curved space-time, Classical and Quantum Gravity 36, 205007 (2019).
  * de Oliveira and Schmidt [2019] M. de Oliveira and A. G. Schmidt, Exact solutions of dirac equation on a static curved space–time, Annals of Physics 401, 21 (2019).
  * de Oliveira and Schmidt [2020] M. D. de Oliveira and A. Schmidt, Exact solution for the dirac oscillator in curved spacetime, Physica Scripta 10.1088/1402-4896/ab76f9 (2020).
  * Yagdjian and Galstian [2021] K. Yagdjian and A. Galstian, Fundamental solutions for the dirac equation in curved spacetime and generalized euler-poisson-darboux equation, Journal of Differential Equations 300, 80 (2021).
  * Jentschura and Noble [2013] U. D. Jentschura and J. H. Noble, Nonrelativistic limit of the dirac-schwarzschild hamiltonian: Gravitational zitterbewegung and gravitational spin-orbit coupling, Phys. Rev. A 88, 022121 (2013).
  * Jentschura [2018] U. D. Jentschura, Gravitational effects in g-factor measurements and high-precision spectroscopy: Limits of einstein’s equivalence principle, Physical Review A 98, 10.1103/physreva.98.032508 (2018).
  * Jentschura [2013] U. D. Jentschura, Gravitationally coupled dirac equation for antimatter, Phys. Rev. A 87, 032101 (2013).
  * Note [1] If the Lagrangian density of the free weak gravitational field $h_{\mu\nu}$ were included into the overall Lagrangian density $L$.
  * Freese [2022] A. Freese, Noether’s theorems and the energy-momentum tensor in quantum gauge theories, Physical Review D 106 (2022).
  * Bunney and Gradoni [2022] C. R. Bunney and G. Gradoni, Electromagnetism in curved space–time: Coupling doppler shifts and gravitational redshifts, IEEE Antennas and Propagation Magazine 64, 40–51 (2022).
  * Weinberg [1972] S. Weinberg,  _Gravitation and Cosmology: Principles and Applications of the General Theory of Relativity_ (John Wiley and Sons, New York, 1972).
  * Feynman [2002] R. P. Feynman,  _Feynman lectures on gravitation_ , edited by F. B. Morinigo, W. G. Wagner, and B. Hatfield (Westview Press, 2002).
  * Baker  _et al._ [2021] M. R. Baker, N. Kiriushcheva, and S. Kuzmin, Noether and hilbert (metric) energy-momentum tensors are not, in general, equivalent, Nuclear Physics B 962, 115240 (2021).
  * Markoutsakis [2019] M. Markoutsakis, Revisiting the belinfante prescription for the energy-momentum tensor of spinor electrodynamics, (2019).


# Metrically Scaled Monocular Depth Estimation through Sparse Priors for
Underwater Robots

Luca Ebner1, Gideon Billings2, Stefan Williams2 1 Luca Ebner is with the
Robotic Systems Lab, ETH Zurich and with the Australian Centre for Field
Robotics, University of Sydney<EMAIL_ADDRESS>Gideon Billings and Stefan
Williams are with the Australian Centre for Field Robotics, University of
Sydney

###### Abstract

In this work, we address the problem of real-time dense depth estimation from
monocular images for mobile underwater vehicles. We formulate a deep learning
model that fuses sparse depth measurements from triangulated features to
improve the depth predictions and solve the problem of scale ambiguity. To
allow prior inputs of arbitrary sparsity, we apply a dense parameterization
method. Our model extends recent state-of-the-art approaches to monocular
image based depth estimation, using an efficient encoder-decoder backbone and
modern lightweight transformer optimization stage to encode global context.
The network is trained in a supervised fashion on the forward-looking
underwater dataset, FLSea. Evaluation results on this dataset demonstrate
significant improvement in depth prediction accuracy by the fusion of the
sparse feature priors. In addition, without any retraining, our method
achieves similar depth prediction accuracy on a downward looking dataset we
collected with a diver operated camera rig, conducting a survey of a coral
reef. The method achieves real-time performance, running at 160 FPS on a
laptop GPU and 7 FPS on a single CPU core and is suitable for direct
deployment on embedded systems. The implementation of this work is made
publicly available at https://github.com/ebnerluca/uw_depth.

###### Index Terms:

Marine Robotics, Computer Vision for Automation, Deep Learning for Visual
Perception

##  I Introduction

Autonomous underwater vehicles often rely on visual based sensing for
localization. For these systems, it is generally sufficient to build a sparse
representation of the scene, using landmarks or detected feature points to
optimize the vehicle pose estimates [1]. However, some tasks require a vehicle
to operate near or interact with the environment, such as conducting close
range imaging surveys or performing manipulation tasks. For such applications,
a dense and real-time understanding of the scene is required for safe
autonomous operation of the vehicle to enable obstacle avoidance and
intervention planning. A practical solution to this problem must be robust to
underwater imaging effects (Fig. 1), such as nonlinear attenuation,
backscatter, and inconsistent lighting, must provide metrically accurate depth
measurements, must be computationally efficient for real-time deployment on
embedded compute systems, and ideally should be compact and low-cost for
integration on diverse underwater robotic platforms.

Figure 1: Optical challenges in underwater imagery. Attenuation (left),
reflections (middle) and turbidity (right).

Active-light depth sensors, such as LiDAR [2] or RGB-D cameras [3, 4], are
commonly used in terrestrial applications and provide an intuitive solution
for metrically accurate depth sensing. However, these sensors are challenging
to deploy underwater, due to nonlinear, frequency dependent attenuation of
light in the water column, refraction effects, and the scattering effects of
suspended particulates. Imaging sonars provide an alternative solution that is
robust to underwater lighting effects, and they are used in many marine field
robotics applications [5]. However, they are expensive and provide limited
spatial resolution compared to optical sensors with noisy depth estimates that
make them less suitable for complex intervention and close range inspection
tasks.

Passive stereo cameras provide a low-cost, off-the-shelf solution that is
simple to setup and can provide dense, high-resolution depth measurements of
the scene. However, stereo cameras can be bulky to integrate in underwater
housings with an adequate camera baseline, making them difficult to deploy on
small underwater systems or on the front of streamlined AUVs with highly
limited space. Passive monocular cameras provide an attractive alternative to
stereo cameras, due to their compactness and being readily integrated on any
underwater platform. For this reason, predicting dense depth from monocular
images for marine robotics has gained more attention in recent years, with
significant advancements made in deep learning approaches.

In this work, we extend the recent advancements in learning based monocular
depth estimation to demonstrate a practical approach for metrically accurate
depth predictions that is suitable for deployment on diverse underwater
platforms and is transferable to different camera systems without retraining,
special calibration, or parameter tuning. In particular, we make the following
contributions:

  * •

An extension of a state-of-the-art lightweight network for dense depth
prediction from monocular images to fuse triangulated feature points into the
prediction stages through a dense parameterization of these sparse priors,
improving the prediction accuracy and providing metric scale constraints.

  * •

Training and evaluation of the method on the FLSea [6] dataset, demonstrating
significantly improved performance through fusion of the sparse priors.

  * •

Evaluation of the method on an additional image dataset from a coral reef
survey we collected off of Lizard Island, Australia, with a downward facing
diver operated camera rig. No images from this dataset or camera rig were used
in the training of the method, demonstrating that the method generalizes well
to different camera systems and environments.

##  II Related Work

###  II-A Monocular Underwater Depth Estimation

Some early approaches to underwater depth estimation from monocular images
were driven by the need to enhance underwater images [7]. These methods aimed
to mitigate the challenges posed by underwater lighting and attenuation
effects, effectively transforming underwater images to resemble those taken in
a terrestrial environment. To achieve this, researchers constructed models
that explicitly account for the physics underlying underwater illumination and
image formation. These methods, known as image formation models, incorporate
coefficients to represent attenuation and other relevant water column
properties. More recently, supervised deep learning approaches have shown
great potential for learning image formation models and predicting depth maps
from monocular images [8]. However, image formation models suffer from poorly
constrained depth estimates and sensitivity to difficult imaging conditions
such as turbidity and reflections.

Given the challenges and expense of collecting large-scale underwater datasets
in diverse environments for training supervised models, another line of
research has focused on generating realistic synthetic underwater datasets to
train depth prediction models without the need for real images [9, 10]. While
a promising avenue of research, the synthesis of underwater data is not yet
able to fully capture the realistic formation of underwater images, leading to
degraded performance on real data.

Recent work has achieved state-of-the-art performance through transfer
learning, where a monocular depth estimation model is trained on large-scale
terrestrial data and then fine tuned with a smaller underwater dataset. [11]
provides a survey of these recent works, where popular adopted models include
Monodepth2 [12], AdaBins [13], and U-Net [14] based architectures.
Additionally, by incorporating domain specific loss formulations, the
performance of the underlying models can be further improved for underwater
environments [11]. Our method extends the work of AdaBins [13] and UDepth [11]
in order to boost the prediction accuracy and provide metrically accurate
scale through the fusion of sparse feature priors in the network architecture.

###  II-B Sparse Measurement Fusion

An inherent problem of monocular camera systems is scale ambiguity. While
supervised deep learning models are able to predict metric depth from
monocular RGB image inputs without additional measurements, such models suffer
from poor generalizability when the data distribution of the real world
application does not match the distribution of the training data or the camera
parameters change. Sparse depth priors from external measurements can be used
to deliver a valuable depth prior and guidance signal for estimation in
accurate metric scale, as surveyed in [15]. We provide a brief review here of
learning based approaches for sparse prior fusion in depth prediction.

Categorized as early fusion, the first learning based methods simply
concatenated the sparse depth maps and RGB image together as the network input
[16], or convolved them in the first few layers of the model. In recent works,
late fusion techniques have been developed with more complex architectures
such as dual-encoder [17] or double encoder-decoder [18] models to better
encode the sparse depth prior into the RGB image feature space. As a result,
late fusion techniques generally outperform early fusion models at the cost of
increased complexity. To cope with the high sparsity of the depth input, these
methods often use either binary or continuous validity masks to guide the
convolution operations to the relevant parts of the input prior. In the most
recent approaches, researchers have begun to incorporate surface normal
representations [19], affinity [20], and residual depth maps [21] into early,
late or hybrid fusion architectures.

Many deep learning based methods for depth estimation from monocular images
treat their problem in a one-shot fashion. At each inference step a new image
is fed into the network to predict depth with no connection between subsequent
frames or timesteps. In robotics however, these image inputs usually come from
video streams, where there is a large correlation between sequential frames.
Notably, many intervention robots are already capable of accurate localization
through off-the-shelf visual-inertial methods like ORB-SLAM3 [22], which store
knowledge about the environment in the form of sparse triangulated visual
feature points. As proposed in methods like [23], such sparse depth priors can
be used in fusion with corresponding RGB images to provide metric scale cues
to a learning method, solving the problem of scale ambiguity and improving the
inference accuracy. We draw inspiration from these prior works in our approach
to fusing sparse feature based priors into the depth prediction network
architecture, following a late fusion approach.

##  III Method

###  III-A Depth Prior Parameterization

To overcome the problem of scale ambiguity and to recover the global scale of
the scene, we develop an approach to include sparse depth measurements of the
scene in the form of visual features as input to the depth prediction network.
We adapt the densified parameterization suggested by [23], which enables
fusion of sparse priors with arbitrary density.

####  III-A1 Sparse Prior Generation

By using the video stream from a robotic system, we can track and triangulate
visual keypoints between frames. Popular visual-inertial SLAM methods like
ORB-SLAM3 [22] track and triangulate features over many frames, resulting in
highly accurate priors. As a simplification for our evaluation on the FLSea
[6] dataset, we use the matched visual features between two consecutive frames
and extract the depth value from the available dataset ground truth. These
sparse measurements are then used as the prior guidance signal for the depth
prediction task. The steps to generate a sparse set of prior depth
measurements are as follows:

  1. 1.

The image is divided into equal sized patches and a fixed number of SIFT
keypoints [24] are detected in each patch.

  2. 2.

The detected features are matched between frames by performing bidirectional
nearest neighbor search followed by enforcing an epipolar constraint for
outlier filtering. The epipolar constraint can be described as
$k_{i}^{1}=F^{T}k_{i}^{2}$, where $(k_{i}^{1},k_{i}^{2})$ denotes the $i$-th
keypoint match between frames $(1,2)$ and $F$ is the fundamental matrix.

  3. 3.

For each inlier feature point, depth values are extracted from the ground
truth depth maps at the keypoint location.

For evaluation on the LizardIsland dataset, we use the SLAM method developed
in [25] to generate feature priors from the image sequence, where the feature
positions and depths are projected from the constructed SLAM feature map,
demonstrating the practical performance of the method as it would be deployed
on an underwater vehicle with an online SLAM system.

####  III-A2 Dense Prior Parameterization

Since the amount of available keypoints may vary from image to image, a
generalized representation of constant size and suitable format is desirable
to feed into the network. Inspired by [23], we define two maps $S_{1}(x,y)$
and $S_{2}(x,y)$ to map the sparse set of keypoints $k_{i}$ into two
continuous image representations.

Following the approach of [23], $S_{1}$ is a nearest neighbor interpolation of
the sparse keypoint depths into a full size dense depth map, where every pixel
$(x,y)$ in $S_{1}(x,y)$ takes the depth value $d_{k^{*}}$ of its closest
keypoint $k^{*}$. The formula to compute $S_{1}$ is given as:

$S_{1}(x,y)=d_{k^{*}}\text{, where
}k^{*}=\underset{i}{\arg\min}||(x,y)-(x_{i},y_{i})||_{2}$ (1)

$S_{2}$ is formulated as a probability map to account for the distance of each
pixel to its nearest keypoint. Let $r(x,y|k_{i})$ be the euclidean pixel
distance of point $(x,y)$ to keypoint $k_{i}$. We assume the probability with
regards to the pixel distance to a given keypoint to follow a normal
distribution. The computation of $S_{2}$ then follows as:

$S_{2}(x,y)=\frac{1}{\sigma\sqrt{2\pi}}\exp\left(-\frac{r(x,y|k^{*})^{2}}{2\sigma^{2}}\right),$
(2)

where $r(x,y|k^{*})=\underset{i}{\min}||(x,y)-(x_{i},y_{i})||_{2}$ and
$\sigma$ is the standard deviation which is treated as a tuning parameter. As
a result of empirical testing, we use $\sigma=10$ for all our experiments.

To form the prior input to our model, the two maps $S_{1}$ and $S_{2}$ are
depth-wise concatenated in the form of a two-channel image. This image is then
fed into the network at multiple stages. Figure 2 shows an example of the
parameterized prior inputs.

Figure 2: Dense prior parameterization. RGB (first), ground truth depth data
available from the given dataset (second), the $S_{1}$ nearest neighbor depth
map (third) and the $S_{2}$ prior probability map (fourth), where the
probability is highest (bright spots) at the location of the features.

###  III-B Architecture Design

The proposed network architecture is based on the popular AdaBins [13] model
and the contributions of UDepth [11]. The architecture consists of a
lightweight encoder-decoder backbone, an efficient vision transformer and a
convolutional regression module. Figure 3 shows an overview of the model
architecture.

Figure 3: Model overview with three main components based on AdaBins [13]:
Encoder-decoder, vision transformer and regression. The depth prior is fed
into the model at multiple stages during the decoder as well as in the
beginning of the vision transformer.

####  III-B1 Encoder Decoder Backbone

In the first module of the proposed network, we use a computationally fast
encoder-decoder backbone. Motivated by the contributions of UDepth [11], we
exchange the heavier encoder module of AdaBins [13] with a more efficient
model based on MobileNetV2 [26]. This model is optimized for mobile
architectures and lays the foundation for many applications which need real-
time performance. Additionally, UDepth has demonstrated that this architecture
attains good feature representations for underwater applications. The encoder-
decoder module is constructed in a U-Net fashion [14]: In the encoder, we pass
the input RGB image through all layers of the MobileNetV2 model and extract
the signal at multiple stages for later use as skip connections. The decoder
then consists of several upsampling layers which combine the current signal
with skip connections from the encoder layers combined with the appropriately
scaled depth parameterization.

####  III-B2 Vision Transformer

While the preceding encoder-decoder block provides spatial feature information
at multiple resolutions, the vision transformer component refines the results
with global context.

Figure 4: Vision Transformer Structure based on AdaBins [13]. The feature
image coming from the preceeding encoder-decoder module and depth
parameterization are spatially divided into a number of patches of size
$p\times p$. Using a convolutional block we have a set of patch embeddings as
input sequence for the transformer encoder. After the encoder, one output
embedding is used as input for a multilayer perceptron (MLP) head to predict
the bin width logits and additionally range $r$, from which we then compute
the metric bin widths via Equation 3. A longer subset of output embeddings is
used as attention query kernels for forming the output attention maps via dot
product.

While vision transformers in general are computationally expensive [27],
AdaBins introduced a lightweight vision transformer that we use with some
modifications [13]. Instead of estimating absolute depth values directly, they
divide the depth range into $n$ bins with adaptive size and predict the
Softmax classification score for every pixel to belong to each of the bins.
While we adopt this mechanism for estimating adaptive bin widths and range
attention maps, our modified architecture additionally predicts the depth
range $r$ for each image. That is, we estimate the bin widths $b_{i}$ by
multiplying the normalized bin widths by the estimated range rather than a
fixed upper bound, as was implemented in [13]. In our testing, we found that
this modification to the network improves performance on inputs which are at
different ranges than the training data.

To obtain the normalized bin widths $\bar{b}_{i}$, the adaptive bin logits
$\tilde{b}_{i}$ output by the transformer are normalized such that they sum up
to 1. For strictly positive bin widths, a small positive value
$\epsilon=10^{-3}$ is added to all logits. The formulation for computing the
bin widths $b_{i}$ is as follows:

$b_{i}=r\cdot\bar{b}_{i}=r\cdot\frac{\tilde{b}_{i}+\epsilon}{\sum_{j}^{n}(\tilde{b}_{j}+\epsilon)}\text{
, where }i=1,2,...,n$ (3)

The bin width logits $\tilde{b}_{i}$ and range $r$ are predicted through a
multilayer perceptron (MLP) head with $n+1$ outputs ($n$ bins plus 1 for range
$r$) from the first transformer output embedding. Figure 4 shows a
visualization of our modified vision transformer module.

####  III-B3 Convolutional Regression

The last stage of the network forms the depth image prediction from the bin
centers and range attention maps output by the vision transformer module. The
bin classification scores are computed following the approach of AdaBins [13],
where the range attention maps from the vision transformer module are $1\times
1$ convolved, followed by a Softmax activation to map the output logits into a
set of $n$ probabilities for each pixel, where $n$ is the number of bins. The
final depth image prediction $\hat{d}(x,y)$ is then formed as the linear
combination of the probabilities $p_{i}(x,y)$ and bin centers $c_{i}$, as
formulated in Equation 4:

$\hat{d}(x,y)=\sum_{i}^{n}c_{i}\cdot p_{i}(x,y)$ (4)

###  III-C Loss Functions

The training loss is formulated as a combination of different loss functions,
providing different constraints on the network predictions. In the following,
we denote $\hat{d}$ as the predicted depth and $d$ as the available ground
truth.

####  III-C1 Root Mean Squared Error

The traditional RMSE loss function directs the network to learn an exact
metric scale prediction. This loss is formulated as

$\mathcal{L}_{\text{RMSE}}(\hat{d},d)=\sqrt{\frac{1}{N}\sum_{i}^{N}(\hat{d}_{i}-d_{i})^{2}}$
(5)

####  III-C2 Scale Invariant Logarithmic Loss

For robotic intervention tasks errors are often calculated in logarithmic
space [28], which penalizes errors at close range and becomes more forgiving
at greater distances. Motivated by [13, 11], we utilize a parameterized
variation of the Scale Invariant Logarithmic Loss (SILog Loss)[29], which
allows us to balance the learning focus between estimating accurate metric
scale and relative depth:

$\mathcal{L}_{\text{SILog}}(\hat{d},d)=\beta\sqrt{\frac{1}{N}\sum_{i}^{N}g_{i}^{2}-\frac{\lambda}{N^{2}}(\sum_{i}^{N}g_{i})^{2}}$
(6)

where $g_{i}=\log\hat{d}_{i}-\log d_{i}$ and $\lambda\in[0,1]$. As proposed by
[13], we use $\lambda=0.85$ and $\beta=10$ in all of our experiments.

####  III-C3 Chamfer Distance Loss

To guide the estimation of adaptive bin sizes $b_{i}$ (described in Subsection
III-B), we utilize a regularizer which encourages the distribution of bin
centers to follow the actual depth distributions of a given input image.
Following [13], we use the Chamfer Distance Loss first introduced by [30]:

$\displaystyle\mathcal{L}_{\text{Chamfer}}(c,d)=\sum_{c_{i}\in
c}\min_{d_{j}\in d}||d_{j}-c_{i}||_{2}^{2}+\sum_{d_{j}\in d}\min_{c_{i}\in
c}||d_{j}-c_{i}||_{2}^{2}\>,$ (7) $\displaystyle\text{where
}i=1,2,...,n,\;j=1,2,...,N$

####  III-C4 Learning Objective

As our learning objective, we use a weighted combination of the losses
$\mathcal{L}_{\text{RMSE}}$, $\mathcal{L}_{\text{SILog}}$ and
$\mathcal{L}_{\text{Chamfer}}$, with $\lambda_{1}=0.3$, $\lambda_{2}=0.6$ and
$\lambda_{3}=0.1$.

$\mathcal{L}_{\text{Objective}}=\lambda_{1}\cdot\mathcal{L}_{\text{RMSE}}+\lambda_{2}\cdot\mathcal{L}_{\text{SILog}}+\lambda_{3}\cdot\mathcal{L}_{\text{Chamfer}}$
(8)

##  IV Experiments

###  IV-A Datasets

####  IV-A1 FLSea

Our method is trained and evaluated on the FLSea dataset [6], which features
forward looking imagery close to the sea floor in relatively shallow water
depths less than $10\text{\,}\mathrm{m}$. On average, the maximum distance
range in the available ground truth is around $10\text{\,}\mathrm{m}$. The
publicly available dataset consists of 22’451 image frames with RGB and metric
ground truth depth data at 12 different underwater locations. Of these 12
locations, we use 10 for training and 2 for testing (u_canyon and sub_pier).
FLSea showcases many of the lighting challenges described in Section II. In
many instances, the imagery includes reflections, turbidity and attenuation
all at the same time. Despite providing color corrected images, we use the raw
images both during training as well as testing. Example pairs of RGB and depth
map data from the FLSea dataset are shown in Figure 5.

Obtaining true underwater depth is a very challenging task. As explained in
Section I, sensors such as LiDAR or RGB-D cameras have only very limited
performance due to the physical properties of the water column. For this
reason, the ground truth depth maps of FLSea are generated using the Agisoft
Metashape photogrammetry software [31]. Photogrammetry pipelines such as
Metashape are considered state-of-art solutions for underwater 3D
reconstruction in both research and industry. However, due to their extreme
computational cost, such pipelines can not be deployed in online applications.
Achieving offline photogrammetry accuracy with a real-time method is an
ultimate aim of our research.

In a pre-processing step, we match features between frames and extract sparse
prior points with their depth values for all images in the FLSea dataset. For
this, we follow the procedure described in Subection III-A and store the
keypoints as a list for later usage during training and testing.

####  IV-A2 LizardIsland Dataset

In order to test the proposed method’s generalization to different camera
systems and underwater environments not represented in the training data, we
additionally evaluate the method performance on a dataset collected with a
downward facing diver operated camera rig conducting a survey of a Coral Reef
off of Lizard Island, Australia. The dataset is composed of a stereo image
sequence which was processed through COLMAP [32] to obtain ground truth dense
depth maps. The sequence was also processed through the SLAM method published
in [25] to generate optimized feature maps that were projected back into the
left camera image frame to obtain the sparse depth priors for input into the
network. Example images from this dataset along with the sparse prior
visualizations are shown with the evaluation results in Figure 6, where the
left camera image was used as the monocular input to the network. The survey
dataset includes 2’216 sequential images that were included in the evaluation
of the method performance.

###  IV-B Learning Pipeline

For the implementation of this work we used the PyTorch [33] library. We
initialize our model at random, except for the MobileNetV2 [26] part in our
encoder-decoder module (see Subsection III-B). As our optimizer we choose
AdamW [34] and train all 15.6 M network parameters freely in steps with a
batch size of 6. For the learning rate $lr(t)=lr_{0}\cdot r^{t}$ we use
exponential decay over time, where $lr_{0}=0.0001$ is the base learning rate
and $r=0.9$ is the decay rate. The model is trained in a supervised fashion on
training data from FLSea. For training robustness, we apply a series of random
transforms to our inputs: Horizontal flips, color- and brightness scaling, as
well as depth scaling. Throughout all training for these tests, we randomly
select 200 depth priors per image frame. From these sparse priors we form our
dense prior parameterization (see Subsection III-A) and feed it into the
network as illustrated in Figure 3. Note that with an output resolution of
$320\times 240$, the 200 keypoints are extremely sparse and only cover 0.26%
of the total number of pixels. The training loss is then computed between the
prediction outputs and target depth maps by using the learning objective
formulated in Subsection III-C.

Using a NVIDIA GTX 2080 graphics card, the pipeline training took
approximately 1 hour per epoch. The best results were achieved after around 10
hours of training.

###  IV-C Evaluation Metrics

To evaluate the performance of the trained models, we use a variety of
standard error metrics. The error metrics between the predicted $\hat{d}$ and
ground truth depth $d$ are calculated as follows: For its straightforward
interpretation, we choose standard RMSE in linear space (Eq. 9). Following the
intuition in Subsection III-C, we further evaluate on RMSE in logarithmic
space (Eq. 10) to investigate emphasized errors at close range. We also
compare this error against its scale invariant counterpart (Eq. 11) to analyze
the accuracy of the overall estimated scene scale.

$\text{RMSE}_{\text{lin}}(\hat{d},d)=\sqrt{\frac{1}{N}\sum_{i}^{N}(\hat{d}_{i}-d_{i})^{2}}\\\
$ (9)
$\text{RMSE}_{\text{log}}(\hat{d},d)=\sqrt{\frac{1}{N}\sum_{i}^{N}(\log\hat{d}_{i}-\log
d_{i})^{2}}$ (10)
$\text{RMSE}_{\text{silog}}(\hat{d},d)=\sqrt{\frac{1}{N}\sum_{i}^{N}(\log\hat{d}_{i}-\log
d_{i}+\alpha(\hat{d},d))^{2}},$ (11)

where $\alpha(\hat{d},d)=\frac{1}{N}\sum_{i}^{N}(\log d_{i}-\log\hat{d}_{i})$
is the term that makes the error scale invariant [29].

Furthermore, we analyze the Mean Absolute Relative Error to gain insight into
the relative accuracy of our prediction:

$\text{MARE}(\hat{d},d)=\frac{1}{N}\sum_{i}^{N}\frac{|\hat{d}_{i}-d_{i}|}{|d_{i}|}$
(12)

###  IV-D Results

####  IV-D1 FLSea

We evaluate the performance of the network on the test split of the FLSea [6]
dataset and analyze how the prediction improves by using sparse priors. As
when training, we use 200 randomly selected sparse priors per image which are
stored from our pre-processing step. Figure 5 shows a visualization of our
predictions and Table I presents the evaluated error metrics, cut off at
different ranges to provide insight into near depth and far depth prediction
accuracy.

Figure 5: Depth prediction examples on the FLSea dataset. RGB input (first row), ground truth depth map (second row), predicted depth map (third row) and absolute error (fourth row). Despite various optical challenges such as attenuation, reflections and turbidity our approach yields stable predictions. Table I: Evaluation metrics of the network tested on our test split of the FLSea dataset. The usage of sparse priors versus without shows significant improvements all across the board. When using sparse priors, the metrics for RMSE in log versus silog space are very similar, indicating a good estimation of our predicted depth range. Range | Priors | RMSE | RMSE | RMSE | MARE
---|---|---|---|---|---
|  | (linear) | (Log) | (SILog) |
Full range | 0 | 0.596 | 0.171 | 0.127 | 0.137
| 200 | 0.372 | 0.089 | 0.087 | 0.037
| Improvement | 37.6% | 48.0% | 31.5% | 73.0%
$d<$5\text{\,}\mathrm{m}$$ | 0 | 0.390 | 0.157 | 0.102 | 0.134
| 200 | 0.167 | 0.057 | 0.055 | 0.031
| Improvement | 57.2% | 63.7% | 46.1% | 76.9%
$d<$1\text{\,}\mathrm{m}$$ | 0 | 0.263 | 0.254 | 0.049 | 0.281
| 200 | 0.040 | 0.042 | 0.032 | 0.029
| Improvement | 84.8% | 83.5% | 34.7% | 89.7%

The results of Table I demonstrate significantly improved depth prediction
accuracy for all depth ranges and across all evaluation metrics through the
fusion of sparse feature priors into the network. It is important to reiterate
that the ground truth depth maps of the FLSea [6] dataset were computed
through a photogrammetric 3D reconstruction pipeline, so these metrics
represent the ability of our method to reproduce these photogrammetry results.
However, as discussed in Subsection IV-A, such pipelines are recognised as
state-of-art solutions for both research and industry.

####  IV-D2 LizardIsland

We evaluated the performance of the method on the LizardIsland dataset, with
the model trained only on the FLSea dataset. Figure 6 shows visualizations of
the predictions for a few samples from this dataset, and Table II gives the
quantitative evaluation of the depth prediction accuracy. In addition to
evaluating the predicted depth map accuracy, we also evaluate the accuracy of
the nearest neighbor prior depth maps, using the same evaluation metrics. The
poor accuracy of the raw nearest neighbor prior maps indicates that they are
too coarse of an estimate to be useful for dense depth prediction by
themselves. However, the network predictions achieve similar accuracy on the
LizardIsland sequence as they do on the test sequences of the FLSea dataset,
demonstrating that the method can effectively generalize to novel underwater
environments and camera systems not included in the training data.

Figure 6: Visualization of prediction results on samples from the LizardIsland dataset. RGB input (first row), ground truth depth map (second row), prior probability distribution map (third row), prior nearest neighbor map (fourth row), predicted depth map (fifth row) and absolute error between the prediction and ground truth depth maps (sixth row). Table II: Evaluation of the proposed method on the LizardIsland dataset, with the model trained only on the FLSea [6] dataset. The evaluated depth range was limited to $5\text{\,}\mathrm{m}$. Mean accuracy of the predicted depth maps (top row) versus standalone input nearest neighbor prior (bottom row). |  | RMSE
---
(linear)
| RMSE
---
(log)
| RMSE
---
(SIlog)
| MARE
---
Prediction | 0.0876 | 0.0430 | 0.0410 | 0.0274
Prior | 1.4790 | 3.9999 | 3.1571 | 0.7230

##  V Conclusion

In this work, we have demonstrated a compact and cost-efficient dense depth
sensing solution for mobile underwater robots. Our method follows a supervised
monocular deep learning approach, where we fuse sparse depth measurements in
order to solve the problem of scale ambiguity and improve the depth prediction
accuracy. Experiments conducted on challenging underwater datasets demonstrate
significantly improved depth prediction performance through fusion of the
sparse priors. With a Root Mean Squared Error of $0.167\text{\,}\mathrm{m}$
($5\text{\,}\mathrm{m}$ range) and $0.040\text{\,}\mathrm{m}$
($1\text{\,}\mathrm{m}$ range) on the FLSea [6] dataset, we achieve promising
accuracies for using this method in underwater intervention and close range
survey applications. However, some intervention tasks may require higher
levels of precision, motivating further research to improve this method.
Considering that our approach requires only a monocular camera and some
knowledge about the robot’s motion, our solution is easy to integrate and
particularly attractive for low-cost and small sized vehicles. Regarding
computational efficiency, our proposed model remains lightweight and achieves
real-time framerates at $160\text{\,}\mathrm{Hz}$ on a NVIDIA GTX 2080 gpu or
$7\text{\,}\mathrm{Hz}$ on an Intel i9-9900K cpu using a single core.

In an outlook for future work, we will explore replacing the naive nearest
neighbor approach of the depth prior parameterization with a smoother
representation or better informed guess, such as propagating the previous
prediction as an input for the current prior guess. In a similar matter, we
would seek to integrate knowledge about the sparse depth priors’ uncertainty
which is often available when using SLAM pipelines. Furthermore, we would like
to incorporate a segmentation for points at infinity as a side task to handle
such areas more robustly.

## References

  * [1] J. A. Placed, J. Strader, H. Carrillo, N. Atanasov, V. Indelman, L. Carlone, and J. A. Castellanos, “A survey on active simultaneous localization and mapping: State of the art and new frontiers,” _IEEE Transactions on Robotics_ , 2023.
  * [2] D. McLeod, J. Jacobson, M. Hardy, and C. Embry, “Autonomous inspection using an underwater 3d lidar,” in _2013 OCEANS-San Diego_. IEEE, 2013, pp. 1–8.
  * [3] H. Lu, Y. Zhang, Y. Li, Q. Zhou, R. Tadoh, T. Uemura, H. Kim, and S. Serikawa, “Depth map reconstruction for underwater kinect camera using inpainting and local image mode filtering,” _IEEE Access_ , vol. 5, pp. 7115–7122, 2017.
  * [4] S. T. Digumarti, G. Chaurasia, A. Taneja, R. Siegwart, A. Thomas, and P. Beardsley, “Underwater 3d capture using a low-cost commercial depth camera,” in _2016 IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2016, pp. 1–9.
  * [5] K. Sun, W. Cui, and C. Chen, “Review of underwater sensing technologies and applications,” _Sensors_ , vol. 21, no. 23, p. 7849, 2021.
  * [6] Y. Randall and T. Treibitz, “Flsea: Underwater visual-inertial and stereo-vision forward-looking datasets,” 2023.
  * [7] J. Zhou, T. Yang, and W. Zhang, “Underwater vision enhancement technologies: A comprehensive review, challenges, and recent trends,” _Applied Intelligence_ , vol. 53, no. 3, pp. 3594–3621, 2023.
  * [8] P. Hambarde, S. Murala, and A. Dhall, “Uw-gan: Single-image depth estimation and image enhancement for underwater images,” _IEEE Transactions on Instrumentation and Measurement_ , vol. 70, pp. 1–12, 2021.
  * [9] Q. Zhao, Z. Zheng, H. Zeng, Z. Yu, H. Zheng, and B. Zheng, “The synthesis of unpaired underwater images for monocular underwater depth prediction,” _Frontiers in Marine Science_ , vol. 8, p. 690962, 2021.
  * [10] P. G. O. Zwilgmeyer, M. Yip, A. L. Teigen, R. Mester, and A. Stahl, “The varos synthetic underwater data set: Towards realistic multi-sensor underwater data with ground truth,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2021, pp. 3722–3730.
  * [11] B. Yu, J. Wu, and M. J. Islam, “Udepth: Fast monocular depth estimation for visually-guided underwater robots,” 2023.
  * [12] C. Godard, O. Mac Aodha, M. Firman, and G. J. Brostow, “Digging into self-supervised monocular depth estimation,” in _Proceedings of the IEEE/CVF international conference on computer vision_ , 2019, pp. 3828–3838.
  * [13] S. F. Bhat, I. Alhashim, and P. Wonka, “Adabins: Depth estimation using adaptive bins,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021, pp. 4009–4018.
  * [14] N. Siddique, S. Paheding, C. P. Elkin, and V. Devabhaktuni, “U-net and its variants for medical image segmentation: A review of theory and applications,” _Ieee Access_ , vol. 9, pp. 82 031–82 057, 2021.
  * [15] J. Hu, C. Bao, M. Ozay, C. Fan, Q. Gao, H. Liu, and T. L. Lam, “Deep depth completion from extremely sparse data: A survey,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * [16] F. Ma and S. Karaman, “Sparse-to-dense: Depth prediction from sparse depth samples and a single image,” in _2018 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2018, pp. 4796–4803.
  * [17] M. Jaritz, R. De Charette, E. Wirbel, X. Perrotton, and F. Nashashibi, “Sparse and dense data with cnns: Depth completion and semantic segmentation,” in _2018 International Conference on 3D Vision (3DV)_. IEEE, 2018, pp. 52–60.
  * [18] J. Tang, F.-P. Tian, W. Feng, J. Li, and P. Tan, “Learning guided convolutional network for depth completion,” _IEEE Transactions on Image Processing_ , vol. 30, pp. 1116–1129, 2020.
  * [19] J. Qiu, Z. Cui, Y. Zhang, X. Zhang, S. Liu, B. Zeng, and M. Pollefeys, “Deeplidar: Deep surface normal guided depth prediction for outdoor scene from sparse lidar data and single color image,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 3313–3322.
  * [20] X. Cheng, P. Wang, and R. Yang, “Depth estimation via affinity learned with convolutional spatial propagation network,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 103–119.
  * [21] J. Gu, Z. Xiang, Y. Ye, and L. Wang, “Denselidar: A real-time pseudo dense depth guided depth completion network,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 2, pp. 1808–1815, 2021.
  * [22] C. Campos, R. Elvira, J. J. G. Rodríguez, J. M. Montiel, and J. D. Tardós, “Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam,” _IEEE Transactions on Robotics_ , vol. 37, no. 6, pp. 1874–1890, 2021.
  * [23] Z. Chen, V. Badrinarayanan, G. Drozdov, and A. Rabinovich, “Estimating depth from rgb and sparse sensing,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 167–182.
  * [24] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” _International journal of computer vision_ , vol. 60, pp. 91–110, 2004.
  * [25] G. Billings, R. Camilli, and M. Johnson-Roberson, “Hybrid visual slam for underwater vehicle manipulator systems,” _IEEE Robotics and Automation Letters_ , vol. 7, no. 3, pp. 6798–6805, 2022.
  * [26] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018, pp. 4510–4520.
  * [27] K. Han, Y. Wang, H. Chen, X. Chen, J. Guo, Z. Liu, Y. Tang, A. Xiao, C. Xu, Y. Xu _et al._ , “A survey on vision transformer,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 45, no. 1, pp. 87–110, 2022.
  * [28] A. Mertan, D. J. Duff, and G. Unal, “Single image depth estimation: An overview,” _Digital Signal Processing_ , vol. 123, p. 103441, 2022.
  * [29] D. Eigen, C. Puhrsch, and R. Fergus, “Depth map prediction from a single image using a multi-scale deep network,” _Advances in neural information processing systems_ , vol. 27, 2014.
  * [30] H. Fan, H. Su, and L. J. Guibas, “A point set generation network for 3d object reconstruction from a single image,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 605–613.
  * [31] Agisoft, “Metashape photoscan professional,” 2018. [Online]. Available: https://www.agisoft.com/downloads/installer/
  * [32] J. L. Schönberger and J.-M. Frahm, “Structure-from-motion revisited,” in _Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * [33] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga _et al._ , “Pytorch: An imperative style, high-performance deep learning library,” _Advances in neural information processing systems_ , vol. 32, 2019.
  * [34] I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” in _International Conference on Learning Representations_ , 2019. [Online]. Available: https://openreview.net/forum?id=Bkg6RiCqY7