# Motion Degeneracy in Self-supervised Learning of Elevation Angle Estimation
for 2D Forward-Looking Sonar

Yusheng Wang1, Yonghoon Ji2, Chujie Wu1, Hiroshi Tsuchiya3, Hajime Asama1 and
Atsushi Yamashita4 1Y. Wang, C. Wu, H. Asama are with the Department of
Precision Engineering, Graduate School of Engineering, The University of
Tokyo, Japan<EMAIL_ADDRESS>Ji is with the
Graduate School for Advanced Science and Technology, JAIST, Japan.
<EMAIL_ADDRESS>Tsuchiya is with the Research Institute, Wakachiku
Construction Co., Ltd., Japan<EMAIL_ADDRESS>Yamashita
is with the Department of Human and Engineered Environmental Studies, Graduate
School of Frontier Sciences, The University of Tokyo, Japan.
<EMAIL_ADDRESS>
###### Abstract

2D forward-looking sonar is a crucial sensor for underwater robotic
perception. A well-known problem in this field is estimating missing
information in the elevation direction during sonar imaging. There are demands
to estimate 3D information per image for 3D mapping and robot navigation
during fly-through missions. Recent learning-based methods have demonstrated
their strengths, but there are still drawbacks. Supervised learning methods
have achieved high-quality results but may require further efforts to acquire
3D ground-truth labels. The existing self-supervised method requires
pretraining using synthetic images with 3D supervision. This study aims to
realize stable self-supervised learning of elevation angle estimation without
pretraining using synthetic images. Failures during self-supervised learning
may be caused by motion degeneracy problems. We first analyze the motion field
of 2D forward-looking sonar, which is related to the main supervision signal.
We utilize a modern learning framework and prove that if the training dataset
is built with effective motions, the network can be trained in a self-
supervised manner without the knowledge of synthetic data. Both simulation and
real experiments validate the proposed method.

##  I INTRODUCTION

Sonar sensors play a vital role in underwater perception as ultrasound is
invariant to water turbidity and illumination conditions. The next generation
of 2D forward-looking sonar (FLS) can generate images with high resolution and
good quality. These sensors are also small in size and are suitable for use in
commercial-class remotely operated vehicles (ROVs) and autonomous underwater
vehicles (AUVs). Furthermore, they have already been applied to underwater
robotics tasks, such as mosaicking, mapping, and navigation [1, 2, 3]. Owing
to the unique imaging principle, the elevation angle information is missing
during image formation. Therefore, one of the most focused but challenging
topics in FLS research is the retrieval of 3D information.

Early research focused on the sparse 3D reconstruction of acoustic images.
Point and line features were used to represent the 3D model, which is
unintuitive for human comprehension [4]. Recently, researchers have focused on
dense 3D reconstruction based on acoustic images. Generally, these methods can
be classified into multiple- and single-view methods. Multi-view methods use
numerous images to generate 3D models. They usually require a large number of
viewpoints, making it necessary to hover around the target using underwater
vehicles [5, 6] or rotate the camera with motors [2]. Single-view methods
utilize shadow information or model ultrasound propagation for 3D
reconstructions [7]. Such methods only work under ideal conditions and are not
general and robust. Recent deep learning-based methods have shown promising
results for the single-view missing dimension estimation problem [8, 9]. The
network in [9] was trained in a supervised manner, which required 3D
supervision. Because acquiring ground-truth information for underwater scenes
can be difficult due to the limitations of sensing techniques, a self-
supervised training method would be highly valuable. Although in [8], the
missing dimension estimation network can be learned in a self-supervised
manner, it requires pretraining using synthetic images of a similar target
with ground truth labels, which limits the application of the method because
it may require 3D model of a similar scene, and the preparation of synthetic
data also requires considerable effort. Self-supervised learning of elevation
angle estimation for FLS without the help of simulation remains an open
problem.

Figure 1: An example of motion degeneration. (a) The 2D forward-looking sonar
moves along the $x$-axis with a small motion $t_{x}$. (b) All the 2D pixels in
the acoustic image move $-t_{x}$ in the $x$-axis. The 2D pixel displacement is
independent of the 3D structure. Such data may deteriorate the self-supervised
learning process of elevation angle estimation.

The aim of this study was to solve the aforementioned problem. Self-supervised
learning of elevation angle estimation uses two adjacent frames from the
acoustic video, namely, the target view image $I_{t}$ and the source $I_{s}$.
A synthetic target view image $\tilde{I_{t}}$ can be generated from $I_{s}$,
motion $M_{t\rightarrow s}$, and estimated pixel-wise target view elevation
angle $E_{t}=g(I_{t})$. Here $g(.)$ denotes the neural network.
$\tilde{I_{t}}$ can be expressed as
$\tilde{I_{t}}=\pi(I_{s},E_{t},M_{t\rightarrow s})$, where $\pi(.)$ denotes
the inverse warping process. The basic idea is to optimize the following
equation:

$\operatorname*{argmin}_{E_{t}}\mathcal{L}(I_{t},\tilde{I_{t}}),$ (1)

where $\mathcal{L}$ is the loss function to minimize the difference between
the original target image and the one from view synthesis.

We find that the motion type during dataset collection is crucial, which may
lead to training failure if the motion is improper. As shown in Fig. 1, for
some basic motions such as $x$-axis translation, the displacements of the
pixels are independent of the 3D structure. In other words, an arbitrary
$E_{t}$ may satisfy Eq. (1), which degrades the training of network $g$. To
determine the type of motion that may contribute to training, we first analyze
the motion field of the acoustic images. The motion field provides the
relationship between pixel displacement, pixel position, sensor motion, and
elevation angle. Based on theoretical analysis, we utilize the most efficient
motions to generate the dataset. We then build a learning framework using
modern techniques. For real noisy images, we use a pre-trained network to
generate a signal mask to filter the background noise and multi-path
reflections for better performance.

In summary, our contributions are as follows.

  * •

We propose a self-supervised learning method for FLS missing dimension
estimation. Furthermore, we prove that the knowledge of synthetic data is not
essential.

  * •

We analyze the motion field for acoustic images and choose the effective
motions to train the missing dimension estimation network.

  * •

Extensive experiments including simulations and field experiments were carried
out to verify our method.

##  II Related Works

###  II-A Acoustic Camera 3D Reconstruction

Early research on FLS used point or line features to retrieve the 3D
information of an object. Several methods have been proposed to estimate the
3D position of the features and the pose of the camera simultaneously [4, 10,
11]. However, such sparse representation is not intuitive for human
perception. Feature extraction and data association are also problems for
acoustic images, considering the low signal-to-noise ratio (SNR). Wang et al.
applied a Gaussian process with AKAZE features for terrain reconstruction
[12]. Recent works have also used two acoustic cameras to sense the
environment [13, 14]. However, they still require the extracted features to
achieve stable performance.

Dense 3D reconstruction has been a trend in recent studies. One direct
approach is to use multiple views to construct a 3D model. Akyin et al.
applied space carving to small objects using signal masks [15]. Guerneve et
al. linearized the sonar projection model to an orthogonal projection and
applied min filtering to achieve a carving scheme [5]. Wang et al. used
occupancy mapping with an inverse sensor model to probabilistically carve the
space and update an object [2]. Westman et al. utilized non-line-of-sight for
3D reconstruction [16, 17]. Qadri et al. proposed an implicit neural
representation for surface reconstruction [6]. The aforementioned methods
achieve convincing results but require numerous viewpoints. Although in [18],
attempts have been made to estimate 3D information with 2$\sim$3 viewpoints,
it requires training with ground truth labels, which is challenging in
underwater environments.

There is a need to estimate 3D information with a single image for 3D mapping
and navigation through a fly-through motion. The acoustic camera can be
considered as a confocal configuration of a spotlight and camera. Shape-from-
shading scheme can be applied to such problems. Aykin et al. modeled the
ultrasound propagation and assumed diffuse reflection for underwater objects
[7]. 3D models can be generated from object contours. Westman et al. utilized
a similar scheme for 3D reconstruction of flat continuous surfaces [19]. The
above-mentioned methods have strong assumptions regarding the scene and may
fail owing to the low SNR of the acoustic image.

Deep learning techniques have been applied to estimate 3D information from a
single image. Wang et al. proposed a method for estimating the front depth of
an acoustic image. It can solve the non-bijective correspondence problem [9].
However, the network was trained in a supervised manner, where ground truth
labels were required. Although labels can be acquired by generative
adversarial network (GAN) and a simulator, it is necessary to model a similar
scene in the simulator [20]. Considering the difficulty of acquiring ground
truth labels, self-supervised learning methods have also been proposed.
DeBortoli et al. proposed a learning-based method for elevation angle
estimation from a single acoustic image [8]. The network was first trained in
a supervised manner using synthetic images with ground-truth labels and then
fine-tuned using self-supervised signals for real images. However, it is
unclear whether the network can be trained without pretraining using synthetic
images. In fact, during training, supervised learning and self-supervised
learning were alternately carried out to ensure performance. It is also
necessary for deeper discussions of the problem, such as which motion
contributes to the training process, because FLS faces a severe motion
degeneracy problem. In this study, we analyze the motion field of the FLS and
choose efficient motions to build the dataset for self-supervised training.
Using a state-of-the-art training framework and proper datasets, we train the
network with acoustic video and corresponding motion information only.

###  II-B Monocular Camera Self-supervised Depth Estimation

With the rapid development of deep learning technologies, depth estimation
from a single optical image has shown stable performance. Initially, most of
the methods were fully supervised, requiring ground truth depth during
training [21, 22]. However, this is challenging in real-world settings,
considering the difficulty of acquiring ground truth labels. Garg et al.
proposed a method for training a depth estimation network in a self-supervised
manner [23]. A pair of images, the source, and the target with a known small
camera motion were used to train the network. Zhou et al. also used view
synthesis as the supervision signal but proved that camera motion can also be
learned simultaneously [24]. Pure rotation is a well-known degenerate motion
in perspective cameras. The transformation of the two image frames becomes
homography, where 3D information is no longer necessary. Considering the
projection principle of perspective cameras, pure rotations may not contribute
to depth estimation training [25]. Improving the performance on indoor
datasets with more rotational motion remains an open problem. Some recent
studies have begun to consider pure rotation degeneracy [26]. The FLS faces a
severe but unique motion degeneracy problem that will be addressed and
discussed in this study. We focus here on the estimation of the elevation
angle for a given motion and leave the motion estimation to future work.

##  III Problem Analysis

###  III-A Inverse Warping Signal

The core of self-supervised learning is to determine the optimal $E_{t}$ based
on Eq. (1) and use a network $g(.)$ to learn it. The motion between the two
frames must be small to guarantee sufficient overlap and photometric
constancy. As mentioned above, there are cases in which $E_{t}$ cannot be
solved owing to degenerate motions. To analyze the degenerate motions, we
notice that the self-supervised learning approaches are highly related to the
concept of the motion field. The motion field models pixel displacement when
the sensor has a small movement. Denoting the pixel displacement as
$\frac{ds}{dt}$ and the motion field estimation process as $f(.)$, motion
field estimation can be described as follows.

$\frac{d\bm{s}}{dt}=f(\bm{s},E_{t},M_{t\rightarrow s}),$ (2)

where $\bm{s}$ denotes the position of the 2D pixel. Using
$\frac{d\bm{s}}{dt}$ and $I_{s}$, it is possible to generate the synthetic
target view $\tilde{I_{t}}$. This study uses the motion field model to
investigate the self-supervised learning of the elevation angle estimation
problem.

###  III-B Motion Model

First, we explain the 2D forward-looking sonar projection model, as shown in
Fig. 2. A 3D point in the sonar coordinate system is written as follows.

Figure 2: Projection model and the definition of motion. It can be
equivalently seen as a projection of a 3D point to the $\phi=0$ plane, where
the elevation information is missing. Small translations along $x$, $y$ and
$z$ axes are defined as $t_{x}$, $t_{y}$ and $t_{z}$, respectively. Small
rotations along $x$, $y$ and $z$ axes are defined as $\omega_{x}$,
$\omega_{y}$ and $\omega_{z}$.

$\bm{p_{s}}=\begin{bmatrix}r\cos\phi\cos\theta&r\cos\phi\sin\theta&r\sin\phi\end{bmatrix}^{\top}.$
(3)

The position of the 2D pixel $\bm{s}$ is as follows.

$\bm{s}=\begin{bmatrix}x_{s}&y_{s}\end{bmatrix}^{\top}=\begin{bmatrix}r\cos\theta&r\sin\theta\end{bmatrix}^{\top}.$
(4)

Thus, the relationship between $\bm{p_{c}}$ and $\bm{s}$ can be determined.

$\bm{s}=\frac{1}{\cos{\phi}}\begin{bmatrix}1&0&0\\\
0&1&0\end{bmatrix}\bm{p_{s}}.$ (5)

The displacement $\frac{d\bm{s}}{dt}$ is calculated by differentiating
$\bm{s}$ with respect to $dt$ based on Eq. (5).

$\frac{d\bm{s}}{dt}=\frac{1}{\cos{\phi}}\begin{bmatrix}1&0&0\\\
0&1&0\end{bmatrix}\frac{d\bm{p_{s}}}{dt}+\tan{\phi}\bm{s}\frac{d\phi}{dt},$
(6)

where $\frac{d\phi}{dt}$ can be further represented by
$\frac{d\bm{p_{s}}}{dt}$ as follows.

$\frac{d\phi}{dt}=\frac{1}{r}\begin{bmatrix}-\cos{\theta}\sin{\phi}&-\sin{\theta}\sin{\phi}&\cos{\phi}\end{bmatrix}\frac{d\bm{p_{s}}}{dt}.$
(7)

When there is a small motion $(\bm{t},\bm{\omega})$ of the FLS, the
displacement of the stationary 3D point in FLS coordinates follows rigid body
movement which can be written as follows.

$\frac{d\bm{p_{s}}}{dt}=-\bm{\omega}\times\bm{p_{s}}-\bm{t}.$ (8)

After combining Eq. (6) to Eq. (8), the following relationship between the
displacement and sonar motion can be acquired, which is also known as the
motion field.

$\begin{bmatrix}\frac{dx_{s}}{dt}\\\ \\\
\frac{dy_{s}}{dt}\end{bmatrix}=\begin{bmatrix}-\frac{t_{x}}{\cos{\phi}}-(\frac{t_{z}\sin{\phi}}{r})x_{s}-\omega_{z}y_{s}\\\
+(\frac{\sin{\phi}\tan{\phi}t_{x}}{r^{2}})x_{s}^{2}+(\frac{\sin{\phi}\tan{\phi}t_{y}}{r^{2}}\\\
+\frac{\tan{\phi}\omega_{x}}{r})x_{s}y_{s}+(\frac{\tan{\phi}\omega_{y}}{r})y_{s}^{2}\\\
\\\
-\frac{t_{y}}{\cos{\phi}}-(\frac{t_{z}\sin{\phi}}{r})y_{s}+\omega_{z}x_{s}\\\
+(\frac{\sin{\phi}\tan{\phi}t_{y}}{r^{2}})y_{s}^{2}+(\frac{\sin{\phi}\tan{\phi}t_{x}}{r^{2}}\\\
-\frac{\tan{\phi}\omega_{y}}{r})x_{s}y_{s}-(\frac{\tan{\phi}\omega_{x}}{r})x_{s}^{2}\end{bmatrix}.$
(9)

Here, we conduct a discussion based on ARIS EXPLORER 3000, where the aperture
angle in the elevation angle ranges from [-7∘,7∘]. Then, $\cos{\phi}$ ranges
from [0.9925,1], which is reasonable to approximate to 1. In addition,
$\sin{\phi}$ ranges from [-0.1219,0.1219] and $\tan{\phi}$ ranges from
[-0.1228,0.1228], such that $\sin{\phi}\tan{\phi}$ ranges from [0,0.015], and
the second-order term is approximated to zero. The first-order terms,
$\sin{\phi}$ and $\tan{\phi}$ are maintained. Then, Eq. (9) becomes

$\begin{bmatrix}\frac{dx_{s}}{dt}\\\ \\\
\frac{dy_{s}}{dt}\end{bmatrix}=\begin{bmatrix}-{t_{x}}-(\frac{t_{z}\sin{\phi}}{r})x_{s}-\omega_{z}y_{s}\\\
+(\frac{\tan{\phi}\omega_{x}}{r})x_{s}y_{s}+(\frac{\tan{\phi}\omega_{y}}{r})y_{s}^{2}\\\
\\\ -{t_{y}}-(\frac{t_{z}\sin{\phi}}{r})y_{s}+\omega_{z}x_{s}\\\
-(\frac{\tan{\phi}\omega_{y}}{r})x_{s}y_{s}-(\frac{\tan{\phi}\omega_{x}}{r})x_{s}^{2}\end{bmatrix}.$
(10)

In [3], motion fields were used for motion estimation of FLS according to a
flat surface. Although an equation similar to Eq. (9) was induced, it was
assumed that the second-order terms of the pixel position could be ignored. In
other words, the terms with $x_{s}^{2}$ and $y_{s}^{2}$ were ignored. This may
not hold in many cases because the units of $x_{s}$ and $y_{s}$ are meters. In
fact, especially for $x_{s}$, it may range from several meters to tens of
meters, and the second-order may be a huge value.

###  III-C Basic Motion Influence

In this subsection, the influence of the basic motion on the motion field is
discussed. Specifically, the following motions are discussed:1) $x$-axis,
$y$-axis translations, and $z$-axis rotation; 2) $x$-axis rotation; 3)
$y$-axis rotation; and 4) $z$-axis translation.

####  III-C1 $x$-axis, $y$-axis translations and $z$-axis rotation

By setting the other motion parameters to zero, the displacement caused by
horizontal motion (i.e., $x$-axis, $y$-axis translations and $z$-axis
rotation) can be written as follows.

$\begin{bmatrix}\frac{dx_{s}}{dt}\\\ \\\
\frac{dy_{s}}{dt}\end{bmatrix}=\begin{bmatrix}-{t_{x}}-\omega_{z}y_{s}\\\ \\\
-{t_{y}}+\omega_{z}x_{s}\end{bmatrix}.$ (11)

Apparently, the displacement is independent of the elevation angle $\phi$. In
other words, the elevation angle could not be retrieved from the
aforementioned motion. Notably, horizontal motion is frequently used for
underwater robot navigation, if the imaging plane is parallel to the motion
plane, 3D information cannot be acquired.

####  III-C2 $x$-axis rotation

$x$-axis rotation is a well-known effective motion for 3D reconstruction FLS.
Empirically, a small $x$-axis rotation can generate large displacement. This
study provides a mathematical explanation with the following equation.

$\begin{bmatrix}\frac{dx_{s}}{dt}\\\ \\\
\frac{dy_{s}}{dt}\end{bmatrix}\\!=\\!\begin{bmatrix}(\frac{\tan{\phi}\omega_{x}}{r})x_{s}y_{s}\\\
\\\
-(\frac{\tan{\phi}\omega_{x}}{r})x_{s}^{2}\end{bmatrix}\\!=\\!\begin{bmatrix}r\omega_{x}{\tan{\phi}}\cos{\theta}\sin{\theta}\\\
\\\ -r\omega_{x}{\tan{\phi}}{\cos^{2}{\theta}}\end{bmatrix}.$ (12)

As a typical case, where $r=3.5$ m, $\phi=3.5^{\circ}$, and
$\omega_{x}=10^{\circ}$ when the azimuth angle changes from $-15^{\circ}$ to
$15^{\circ}$, the displacement is shown in Fig. 3(a). In this equation,
$\cos^{2}{\theta}$ is relatively large, which makes the displacement obvious.
In this figure, $\rho$ refers to the range resolution equal to 0.003 m and
$\gamma$ is the tangential resolution when $r=3.5$ m. For $\frac{dx_{s}}{dt}$,
a change smaller than $\rho$ cannot be detected in the acoustic image. For
$\frac{dy_{s}}{dt}$, a change smaller than $\gamma$ is difficult to detect.
This may require subpixel optimization for displacement estimation. Notably,
the ARIS EXPLORER 3000 has one of the best resolutions for FLS. The resolution
range for Blueview P900 is approximately 0.0254 m, which is not sufficiently
sensitive to detect some of the displacements for a close range. However,
sensors such as Blueview are designed to be applied to a larger scene, and the
discussions in this section also stand by scaling up the problem with a larger
range and coarser resolution.

####  III-C3 $y$-axis rotation

For $y$-axis rotation, the displacement is dependent on the elevation angle
$\phi$ in the following equation. Theoretically, $\phi$ can be obtained from
the $y$-axis rotation. However, in practice, $y$-axis rotation is not very
efficient [2].

$\begin{bmatrix}\frac{dx_{s}}{dt}\\\ \\\
\frac{dy_{s}}{dt}\end{bmatrix}\\!\\!\\!=\\!\\!\\!\begin{bmatrix}(\frac{\tan{\phi}\omega_{y}}{r})y_{s}^{2}\\\
\\\
-(\frac{\tan{\phi}\omega_{y}}{r})x_{s}y_{s}\end{bmatrix}\\!\\!\\!=\\!\\!\\!\begin{bmatrix}r\omega_{y}{\tan{\phi}}{\sin^{2}{\theta}}\\\
\\\ -r\omega_{y}{\tan{\phi}}\cos{\theta}\sin{\theta}\end{bmatrix}\\!.$ (13)

With the same configuration ($r=3.5$ m, $\phi=3.5^{\circ}$,
$\omega_{y}=10^{\circ}$), Figure 3(b) shows the sensitivity analysis of
$y$-axis rotation. From the results, it is known that the displacement is
quite small, which may make it difficult to detect and influence the 3D
reconstruction performance.

####  III-C4 $z$-axis translation

$z$-axis translation is also considered an effective motion that has been used
for 3D reconstruction.

$\begin{bmatrix}\frac{dx_{s}}{dt}\\\ \\\
\frac{dy_{s}}{dt}\end{bmatrix}=\begin{bmatrix}-(\frac{t_{z}\sin{\phi}}{r})x_{s}\\\
\\\
-(\frac{t_{z}\sin{\phi}}{r})y_{s}\end{bmatrix}=\begin{bmatrix}-{t_{z}\sin{\phi}}\cos{\theta}\\\
\\\ -{t_{z}\sin{\phi}}\sin{\theta}\end{bmatrix}.$ (14)

(a)  (b)

(c)

Figure 3: Sensitivity analysis. (a) The displacement caused by $\omega_{x}$,
where $r=3.5$ m, $\phi=3.5^{\circ}$, and $\omega_{x}=10^{\circ}$. (b) The
displacement caused by $\omega_{y}$, where $r=3.5$ m, $\phi=3.5^{\circ}$, and
$\omega_{y}=10^{\circ}$. (c) The displacement caused by $t_{z}$, where $r=3.5$
m, $\phi=3.5^{\circ}$, and $t_{z}=0.1745$ m.

For the sensitivity analysis, with a configuration of $r=3.5$ m,
$\phi=3.5^{\circ}$, and $t_{z}=0.1745$ m, the results are shown in Fig. 3(c).
The value of $\cos{\theta}$ in Eq. (14) is quite large, which makes the
$z$-axis translation an effective motion.

The following conclusions can be drawn.

  * •

Basic motions $x$-axis rotation and $z$-axis translation are sensitive to
$\phi$, which are effective for tasks like 3D reconstruction.

  * •

The horizontal motions $t_{x},t_{y},\omega_{z}$ are independent to $\phi$ in
motion field, which do not contribute to 3D reconstruction.

  * •

The displacement from $\omega_{y}$ is small, which may be difficult to be
detected.

For dataset generation, basic motions $x$-axis rotation and $z$-axis
translation are theoretically effective.

##  IV End-to-end Learning

###  IV-A Learning Framework

Figure 4: The illustration of the self-supervised learning framework during
training. For test time, it is possible to estimate the elevation map per
image using EAE-Net.

The learning framework is illustrated in Fig. 4. We name the network for
elevation angle estimation as EAE-Net. We use a UNet-structure with a Sigmoid
layer before the final output. The elements in the elevation map are
0$\sim$1\. Denoting the aperture angle in the elevation direction as
$\phi_{\rm aperture}$, the values in the elevation maps are linearly mapped to
[$-\frac{\phi_{\rm aperture}}{2}$,$\frac{\phi_{\rm aperture}}{2}$] for further
processing. By specifying the output range, the training process has better
convergence. Because there is no 3D supervision, two adjacent frames from the
acoustic video are used for training. We first estimate the target view
elevation map $E_{t}$ and transfer it to the point cloud. With known motion
$M_{t\rightarrow s}$, the point cloud is transferred to the source
coordinates. By sampling the source image $I_{s}$, it is possible to generate
the synthetic target image $\tilde{I_{t}}$. Bilinear sampling is used to
ensure the process is differentiable. The network can be optimized by
minimizing the difference between $I_{t}$ and $\tilde{I_{t}}$. Implementation
of inverse warping $\pi(.)$ is expressed as Algorithm 1.

Input : Elevation map in target view $E_{t}$, source image $I_{s}$, motion
$M_{t\rightarrow s}$

Output : Synthetic target view image from source image $\tilde{I_{t}}$

1 foreach  _$(r_{t},\theta_{t},\phi_{t})\in E_{t}$_ do

2 Transform the point to the Euclidean coordinates $\bm{p_{t}}$ using Eq. (3).

3 $\bm{p_{s}}=M_{t\rightarrow s}\bm{p_{t}}$.

4 Project $\bm{p_{s}}$ to the image coordinates by generating
$(r_{s},\theta_{s})$ pairs.

5 Calculate the intensity $i_{s}$ of each point using bilinear sampling.

6 $\tilde{I_{t}}$($r_{t},\theta_{t}$) = $i_{s}$.

7

8return $\tilde{I_{t}}$

Algorithm 1 View synthesis in FLS

###  IV-B Loss Functions

To compare the synthetic target image $\tilde{I_{t}}$ and the real target
image $I_{t}$, L1 and SSIM loss are used. Together they are called the
reconstruction loss $\mathcal{L}_{r}$.

$\mathcal{L}_{r}=\beta\mathcal{M}_{t}(1-{\rm
SSIM}(I_{t},\tilde{I_{t}}))+(1-\beta)\mathcal{M}_{t}||I_{t}-\tilde{I_{t}}||_{1},$
(15)

where $\beta$ is a hyperparameter that balances the two losses, and
$\mathcal{M}_{t}$ is the mask of the target image. We set $\beta$ as 0.3.

Smoothness loss $\mathcal{L}_{s}$ is implemented on the elevation maps in the
target view ${E_{t}}$. The edge-ware smoothness loss is expressed as follows.
It is assumed that the regions between the sharp edges are smooth. This is to
smooth the elevation map when there is not much texture in the target view.

$\mathcal{L}_{s}=|{\partial r}{\mathcal{M}_{t}E_{t}}|e^{-|{\partial
r}I_{t}|}+|{\partial\theta}{\mathcal{M}_{t}E_{t}}|e^{-|{\partial\theta}I_{t}|}.$
(16)

The total loss can be expressed as follows.

$\mathcal{L}=\lambda_{r}\mathcal{L}_{r}+\lambda_{s}\mathcal{L}_{s},$ (17)

where $\lambda_{r}$ and $\lambda_{s}$ are weights to balance the losses. We
set $\lambda_{r}$ to 2 and $\lambda_{s}$ to 1 here.

###  IV-C Signal Mask

For real images, the background noise and multi-path reflections may influence
the training process. This study focuses more on the geometric process during
self-supervised learning and the influence of noise should be eliminated. For
most 3D reconstruction works, the signal mask is essential and acquired by
methods like binarization and region detection [2, 19]. This study uses a
direct but efficient method. We manually label the region with informative
signals, and the remaining pixels are considered to be noise. By creating a
small dataset with real image and signal mask pairs, it is possible to train a
network for binary segmentation. We use a UNet structure as the network and
binary cross entropy (BCE) with Dice loss to train the network [27].

##  V Experiment

###  V-A Simulation Experiment

####  V-A1 Dataset Generation

We used our simulator in Blender to build the synthetic
dataset***https://github.com/sollynoay/Sonar-simulator-blender. Here, we
synthesized scenes of seabed terrain using A.N.T. Landscape add-on in Blender
using hetero noise. Because the elevation angle estimation problem potentially
suffers from the non-bijective 2D-3D correspondence problem [9], one pixel may
correspond to multiple ground truth elevation angle values. To avoid this, in
the synthetic dataset, most of the pixels in the acoustic image correspond to
only one 3D position for a better evaluation. We generated two different
terrains for training, three different terrains for validation, and three
different terrains for test. We generated training datasets with six basic
motions and evaluate them on one test dataset. The configurations of the
datasets are listed in Table I. During training, consecutive three frames were
used: the current frame was the target frame, and the past and the future
frames were the source frames. For the motion between two adjacent frames,
each motion follows uniform distribution and the scope is shown in Table I.
Here, $\omega_{y}$ motion is smaller compared to the other rotational motions
because with a larger $\omega_{y}$ value, the overlap of the consecutive
frames may be insufficient.

TABLE I: Basic motion datasets

Dataset | Training (triplet) | Training (Motion) | Val | Test
---|---|---|---|---
$t_{x}$ | 3,000 | 8$\sim$12 [cm] | 1,500 | 1,500
$t_{y}$ | 3,000 | 8$\sim$12 [cm]
$t_{z}$ | 3,000 | 8$\sim$12 [cm]
$\omega_{x}$ | 3,000 | 5$\sim$10 [deg]
$\omega_{y}$ | 3,000 | 2$\sim$4 [deg]
$\omega_{y}$ | 3,000 | 5$\sim$10 [deg]

####  V-A2 Implementation

This method was implemented using PyTorch. The network was trained and tested
using an NVIDIA GeForce RTX 3090. We set the learning rate to 0.0005 and batch
size to 4. We used the Adam optimizer and trained the network for 15 epochs
for each dataset.

####  V-A3 Metrics

To evaluate the results, we used the mean absolute error (MAE) of the
elevation map. We also transferred the elevation map into point cloud and used
the chamfer distance (CD) for the evaluation. The ground truth point cloud is
from the entire sensible region in the case of the non-bijective 2D-3D
correspondence problem [9].

${\rm
MAE}=\frac{\beta}{HW}\times\sum_{i=1}^{H}\sum_{j=1}^{W}|\hat{D}(i,j)-D(i,j)|,$
(18) ${\rm CD}=\frac{\nu}{S_{1}}\sum_{x\in S_{1}}\min_{y\in
S_{2}}||x-y||_{2}^{2}+\frac{\nu}{S_{2}}\sum_{y\in S_{2}}\min_{x\in
S_{1}}||x-y||_{2}^{2},$ (19)

where $\beta$ and $\nu$ are scale parameters, which refer to 1,000 and 500,
respectively. $S_{1}$ and $S_{2}$ refer to the two point sets. We also used a
percentage metric that measures the overall performance of accuracy and
completeness with an error distance smaller than a threshold as the $f$-score
[28]. For MAE and CD, the results are the smaller the better. The opposite is
true for $f$-score.

####  V-A4 Results

Figure 5: Visualization examples of the elevation angle estimation results.
Color refers to the elevation angle. Images from left to right: (a) input
image, (b) ground truth, (c) supervised learning results, (d) results from
$t_{x}$ dataset, (e) results from $t_{y}$ dataset, (f) results from $t_{z}$
dataset, (g) results from $\omega_{x}$ dataset, (h) results from $\omega_{y}$
dataset, and (i) results from $\omega_{z}$ dataset. For the self-supervised
method, only $\omega_{x}$ and $t_{z}$ datasets can successfully train the
network. TABLE II: Simulation experiment results

|  | EA | Point cloud
---|---|---|---
|  3D label | MAE [rad] | CD [m] |  $f$-score ($<$1 mm) |  $f$-score ($<$3 mm)
EAE | $\circ$ | 0.0234 | 1.210 | 72.71 | 90.61
A2FNet | $\circ$ | – | 1.784 | 63.52 | 84.77
$t_{x}$ | $\times$ | 0.0956 | 161.300 | 0.06 | 0.18
$t_{y}$ | $\times$ | 0.1086 | 191.566 | 0.02 | 0.08
$t_{z}$ | $\times$ | 0.0361 | 3.437 | 50.29 | 73.95
$\omega_{x}$ | $\times$ | 0.0298 | 1.972 | 59.81 | 83.26
$\omega_{y}$ | $\times$ | 0.1086 | 164.500 | 0.00 | 0.00
$\omega_{z}$ | $\times$ | 0.0981 | 176.321 | 0.00 | 0.00

The quantitative results are shown in Fig. 5. EA refers to elevation angle. We
also compared the proposed method with supervised learning methods. EAE refers
to elevation angle estimation using supervised learning. A2FNet estimates the
front depth map instead of the elevation map [9]. The signal mask in the
simulation experiment is generated by binarization with a threshold. It can be
known that $t_{z}$ and $\omega_{x}$ datasets can help the network learn the
elevation angle estimation process. On the other hand, with other datasets,
the network can barely learn any information. This proves the conclusion in
Section III. $t_{x}$, $t_{y}$, and $\omega_{z}$ datasets perform poorly due to
the motion degeneracy. For $\omega_{y}$ datasets, the results are poor due to
the sensitivity of the sensor. It is unsurprising that supervised learning
with ground truth can generate more accurate results. However, by using
$\omega_{x}$ dataset, the network can achieve the same level as A2FNet. This
indicates that with proper training, self-supervised learning methods can
approach supervised learning. One of the keys is using efficient motion. Since
in terrain datasets, most of the pixels only correspond to one 3D position,
direct elevation angle estimation may outperform A2FNet.

Figure 5 shows examples of elevation angle estimation results. It is quite
obvious that only $t_{z}$ and $\omega_{x}$ datasets can contribute to
training. For other motions, the network may converge to a random value. This
is reasonable because based on our motion field analysis, an arbitrary
elevation map may satisfy Eq. (1), so that the network may perform randomly.
After iterations and with the help of smoothness loss, the output of the
network becomes a random constant. It can be also known that $\omega_{x}$
motion is more effective. The results from the $t_{z}$ dataset may contain
some artifacts. This may be because the $\omega_{x}$ motion cause more
overlaps and the photometric constancy holds better.

###  V-B Real Experiment

####  V-B1 Dataset

(a)  (b)

Figure 6: Real experiment: (a) moving device, and (b) target scene.

We built an $\omega_{x}$ dataset in a large-scale water tank to evaluate the
proposed method. We used ARIS EXPLORER 3000 under 3.0 MHz mode. As shown in
Fig. 6(a), the FLS was moved by a moving device. The FLS was carried to 51
positions. For each position, AR2 rotator was used to generate $\omega_{x}$
motion. The FLS rotates along $x$-axis from -35∘ to 35∘. We used images from
33 positions to train the network, 9 positions for validation, and the other 9
positions for test. In total, we used 1,071 images for training, 283 images
for validation, and 294 images for test. We also used triplets to train the
network. Denoting the roll angle for the target frame as $\alpha^{\circ}$, the
roll angles for the two source frames were approximately $(\alpha-10)^{\circ}$
and $(\alpha+10)^{\circ}$, respectively. The target scene is illustrated in
Fig. 6(b). The 3D ground truth of objects in the scene was measured in the
land environment [18]. They were set up in the water tank with a known
configuration so it is possible to acquire the CAD model of the scene. It is
also necessary to know the relative pose between the scene and the FLS. The
initial value is measured by a diver using a ruler and the accurate result is
obtained by comparing the synthetic image with the real image in the edge
domain [29]. After acquiring one pose accurately, the rest can be calculated
with the help of control input for the moving device and the rotator. To
prepare the signal masks, we manually labeled 164 images in the training set
and trained the mask segmentation network. We set the batch size to 4 and the
learning rate to 0.0003. The network was then trained for 25 epochs. Then, for
all the acoustic images in the dataset, we multiplied the binary mask by the
acoustic image to filter the background noise and multi-path reflection
signals. As shown in Fig. 7, Figure 7(a) is an example of a raw acoustic
image. After filtering the noise using the mask, the image in Fig. 7(b) was
used for further processing. To train the EAE-Net, we used the same parameter
setting as in the simulation experiment.

(a)  (b)

Figure 7: Examples of the acoustic images in the dataset. (a) shows the raw
image where there is noise in the non-signal region. Multi-path reflections
may lie in the shadow region which may influence the learning process. (b) is
the image after filtering by the signal mask.

####  V-B2 Results

The quantitative results of the real experiment are listed in Table III.

TABLE III: Real experiment results

|  | EA | Point cloud
---|---|---|---
|  3D label | MAE†††Ground truth elevation angle for some pixels is not unique. [rad] | CD [m] |  $f$-score ($<$1 mm) |  $f$-score ($<$3 mm)
EAE | $\circ$ | 0.0139 | 0.8450 | 83.43 | 94.43
A2FNet | $\circ$ | – | 0.5385 | 92.67 | 97.68
$\omega_{x}$ w/o SM | $\times$ | 0.0364 | 5.243 | 43.16 | 66.44
$\omega_{x}$ w SM | $\times$ | 0.0246 | 2.729 | 57.42 | 83.18

SM refers to the learned signal mask. For the self-supervised method, we also
discussed the influence of the signal mask. It is known that using an
$\omega_{x}$ dataset, the network can be successfully trained without
pretraining on the synthetic images. Although not as good as supervised
learning results, self-supervised learning with a learned signal mask can
generate results with high quality. The $f$-score with a 3 mm threshold is
over 83%, indicating that most of the points have an error smaller than 3 mm.
Considering the resolution in the range direction is 3 mm, this is acceptable
for 3D reconstruction with a single image. If we directly generate the mask by
binarization with a threshold, the mask is noisy and the multi-path signals
cannot be filtered. The result may become slightly worse than that with a
learned mask. A2FNet here exhibits better performance because, in the real
dataset, there are more cases where one pixel corresponds to multiple 3D
positions. The MAE metric here is only for reference because there are
multiple ground truth elevation maps. For point cloud evaluation, the ground
truth point cloud is the entire sensible region [9].

(a)  (b)  (c)  (d)

Figure 8: Estimation results of Fig. 7 in the point cloud: (a) the ground
truth, (b) the result of supervised learning, (c) the result without using the
learned signal mask, the mask is from simple binarization with a threshold,
and (d) the result of the proposed method.

Figure 8 presents the estimation results of Fig. 7 in terms of point cloud. As
shown in Fig. 8(d), the proposed method successfully estimated the 3D
structure of the object. The result of the arc in Fig. 8(d) is better than
that shown in Fig. 8(b) from supervised learning. There will inevitably be
some distortions in the point cloud from self-supervised learning because
there will always be regions that do not overlap between the consecutive
frames. The result in Fig. 8(c) is noisier because of the mask; however, the
3D structure can still be successfully learned.

##  VI Conclusion

In this study, we dug deep into the self-supervised learning of the elevation
angle estimation problem for FLS. It is found that many failure cases may be
caused by the motion degeneracy problem. We analyzed the motion field and
identified efficient motions for network training. We proved that for basic
motions, $t_{z}$ and $\omega_{x}$ motions can contribute to the learning
process. We also showed that pretraining using synthetic data is not a must if
the dataset is properly built. The motion field analysis conclusions may not
only contribute to self-supervised learning through inverse warping signal but
may also be important to simultaneous localization and mapping and multi-view
3D reconstruction problems.

Future work may include auto-filtering degenerate motions from a dataset with
various motions. Signal masks should be further studied. In this study, the
motion between two consecutive frames was considered given. It may also be
worth discussing whether motions can be learned in a self-supervised manner
simultaneously.

## References

  * [1] Natalia Hurtos et al.  “Fourier-based Registration for Robust Forward-looking Sonar Mosaicing in Low-visibility Underwater Environments”  In _J. Field Robot._ 32.1, 2015, pp. 123–151
  * [2] Yusheng Wang et al.  “Acoustic Camera-Based Pose Graph SLAM for Dense 3-D Mapping in Underwater Environments”  In _IEEE J. Oceanic Eng._ 46.3, 2021, pp. 1–19
  * [3] S. Negahdaripour  “On 3-D Motion Estimation From Feature Tracks in 2-D FS Sonar Video”  In _IEEE Trans. Robot._ 29.4, 2013, pp. 1016–1030
  * [4] Ngoc Trung Mai et al.  “3-D Reconstruction of Underwater Object Based on Extended Kalman Filter by Using Acoustic Camera Images”  In _IFAC-Pap._ 50.1, 2017, pp. 1043–1049
  * [5] Thomas Guerneve, Kartic Subr and Yvan Petillot  “Three-dimensional reconstruction of underwater objects using wide-perture imaging SONAR”  In _J. Field Robot._ 35.6, 2018, pp. 890–905
  * [6] Mohamad Qadri, Michael Kaess and Ioannis Gkioulekas  “Neural Implicit Surface Reconstruction using Imaging Sonar”  In _arXiv:2209.08221_ , 2023, pp. 1–8
  * [7] M.. Aykin and S. Negahdaripour  “Modeling 2-D Lens-Based Forward-Scan Sonar Imagery for Targets With Diffuse Reflectance”  In _IEEE J. Oceanic Eng._ 41.3, 2016, pp. 569–582
  * [8] R. DeBortoli, F. Li and G.. Hollinger  “ElevateNet: A Convolutional Neural Network for Estimating the Missing Dimension in 2D Underwater Sonar Images”  In _Proc. IEEE Int. Conf. Intell. Robots Syst._ , 2019, pp. 8040–8047
  * [9] Yusheng Wang et al.  “Elevation Angle Estimation in 2D Acoustic Images Using Pseudo Front View”  In _IEEE Robot. and Autom. Lett._ 6.2, 2021, pp. 1535–1542
  * [10] T.. Huang and M. Kaess  “Towards acoustic structure from motion for imaging sonar”  In _Proc. IEEE Int. Conf. Intell. Robots Syst._ , 2015, pp. 758–765
  * [11] T.. Huang and M. Kaess  “Incremental data association for acoustic structure from motion”  In _Proc. IEEE Int. Conf. Intell. Robots Syst._ , 2016, pp. 1334–1341
  * [12] J. Wang, T. Shan and B. Englot  “Underwater Terrain Reconstruction from Forward-Looking Sonar Imagery”  In _Proc. IEEE Int. Conf. on Robot. Autom._ , 2019, pp. 3471–3477
  * [13] S. Negahdaripour  “Application of Forward-Scan Sonar Stereo for 3-D Scene Reconstruction”  In _IEEE J. Oceanic Eng._ , 2018, pp. 1–16
  * [14] John McConnell, John D. Martin and Brendan Englot  “Fusing Concurrent Orthogonal Wide-aperture Sonar Images for Dense Underwater 3D Reconstruction”  In _Proc. IEEE Int. Conf. Intell. Robots Syst._ , 2020, pp. 1653–1660
  * [15] M.. Aykin and S. Negahdaripour  “Three-Dimensional Target Reconstruction From Multiple 2-D Forward-Scan Sonar Views by Space Carving”  In _IEEE J. Oceanic Eng._ 42.3, 2017, pp. 574–589
  * [16] E. Westman, I. Gkioulekas and M. Kaess  “A Volumetric Albedo Framework for 3D Imaging Sonar Reconstruction”  In _Proc. IEEE Int. Conf. on Robot. Autom._ , 2020, pp. 9645–9651
  * [17] E. Westman, I. Gkioulekas and M. Kaess  “A Theory of Fermat Paths for 3D Imaging Sonar Reconstruction”  In _Proc. IEEE Int. Conf. Intell. Robots Syst._ , 2020, pp. 5082–5088
  * [18] Yusheng Wang et al.  “Learning Pseudo Front Depth for 2D Forward-Looking Sonar-based Multi-view Stereo”  In _Proc. IEEE Int. Conf. Intell. Robots Syst._ , 2022, pp. 8730–8737
  * [19] E. Westman and M. Kaess  “Wide Aperture Imaging Sonar Reconstruction using Generative Models”  In _Proc. IEEE Int. Conf. Intell. Robots Syst._ , 2019, pp. 8067–8074
  * [20] Dingyu Liu et al.  “CycleGAN-based realistic image dataset generation for forward-looking sonar”  In _Adv. Robot._ 35.3-4, 2021, pp. 242–254
  * [21] David Eigen, Christian Puhrsch and Rob Fergus  “Depth Map Prediction from a Single Image using a Multi-Scale Deep Network”  In _Adv. Neural Inf. Process Syst._ , 2014, pp. 2366–2374
  * [22] Fayao Liu, Chunhua Shen, Guosheng Lin and Ian Reid  “Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields”  In _IEEE Trans. Pattern Anal. Mach. Intell._ 38.10, 2016, pp. 2024–2039
  * [23] Ravi Garg, BG Vijay Kumar, Gustavo Carneiro and Ian Reid  “Unsupervised CNN for single view depth estimation: Geometry to the rescue”  In _Proc. Eur. Conf. on Comput. Vis._ , 2016, pp. 740–756
  * [24] Tinghui Zhou, Matthew Brown, Noah Snavely and David G. Lowe  “Unsupervised Learning of Depth and Ego-Motion from Video”  In _Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit._ , 2017, pp. 6612–6619
  * [25] Tom Van Dijk and Guido De Croon  “How Do Neural Networks See Depth in Single Images?”  In _Proc. IEEE Int. Conf. Comput. Vis._ , 2019, pp. 2183–2191
  * [26] Jia-Wang Bian et al.  “Auto-Rectify Network for Unsupervised Indoor Depth Estimation”  In _IEEE Trans. Pattern Anal. Mach. Intell._ 44.12, 2022, pp. 9802–9813
  * [27] Shruti Jadon  “A survey of loss functions for semantic segmentation”  In _Proc. IEEE Conf. Comput. Intell. Bioinform. and Comput. Biol._ , 2020, pp. 1–7
  * [28] Arno Knapitsch, Jaesik Park, Qian-Yi Zhou and Vladlen Koltun  “Tanks and Temples: Benchmarking Large-Scale Scene Reconstruction”  In _ACM Trans. Graph._ 36.4, 2017, pp. 1–13
  * [29] Yusheng Wang et al.  “Simulator-aided Edge-based Acoustic Camera Pose Estimation”  In _OCEANS 2022 - Chennai_ , 2022, pp. 1–4