# MapsTP: HD Map Images Based Multimodal Trajectory Prediction for Automated
Vehicles

Sushil Sharma1,2 1Department of Electronic & Computer Engineering, University
of Limerick, Ireland
2SFI CRT Foundations in Data Science, University of Limerick, Ireland  Arindam
Das1 1Department of Electronic & Computer Engineering, University of Limerick,
Ireland
2SFI CRT Foundations in Data Science, University of Limerick, Ireland  Ganesh
Sistu1 1Department of Electronic & Computer Engineering, University of
Limerick, Ireland
2SFI CRT Foundations in Data Science, University of Limerick, Ireland  Mark
Halton1 and Ciarán Eising1,2 1Department of Electronic & Computer Engineering,
University of Limerick, Ireland
2SFI CRT Foundations in Data Science, University of Limerick, Ireland

###### Abstract

Predicting ego vehicle trajectories remains a critical challenge, especially
in urban and dense areas due to the unpredictable behaviours of other vehicles
and pedestrians. Multimodal trajectory prediction enhances decision-making by
considering multiple possible future trajectories based on diverse sources of
environmental data. In this approach, we leverage ResNet-50 to extract image
features from high-definition map data and use IMU sensor data to calculate
speed, acceleration, and yaw rate. A temporal probabilistic network is
employed to compute potential trajectories, selecting the most accurate and
highly probable trajectory paths. This method integrates HD map data to
improve the robustness and reliability of trajectory predictions for
autonomous vehicles.

Keywords: HDMap Images, MultiModel Trajectory Prediction, Probabilistic
Network, Autonomous Vehicles.

##  1 Introduction

Figure 1: Standard Autonomous Driving Pipeline Overview

Automated vehicles (AVs) are changing the way we think about transportation,
promising to make our roads safer and more efficient [Hancock et al., 2019,
Chehri and Mouftah, 2019, Sharma et al., 2024a]. A key challenge for AVs is
predicting the paths of ego vehicle, other vehicles, pedestrians, and
cyclists. This is essential to avoid collisions [Zhao et al., 2017] and ensure
smooth driving [Huang et al., 2023]. Recently, researchers have been exploring
the use of high-definition (HD) maps combined with various data sources to
improve these predictions [Elghazaly et al., 2023, Bao et al., 2023]. HD maps
offer detailed information about the road, including lane markings, traffic
signs, and road shapes. This information can help AVs better understand their
surroundings and make more accurate decisions. By combining HD maps with
trajectory prediction models, recent studies have shown improved accuracy in
predicting where other road users will go [Guo et al., 2023]. Figure 1 shows
the standard end-to-end pipeline containing essential components for standard
autonomous driving systems.

In this paper, we introduce the model MapsTP, which uses detailed information
from HD maps to enhance trajectory prediction. It also considers sensor data
to understand the different behaviours of road users like cars, pedestrians,
and cyclists. This comprehensive approach is crucial for developing AVs that
can navigate urban city environments safely and efficiently. Our model
predicts ego vehicle trajectories by combining HD map images and dynamic
sensor data. HD maps provide detailed environmental information, while sensors
capture the vehicle’s state. Features are extracted using ResNet-50 and fed
through a Spatio-Temporal Probabilistic Network to estimate multiple potential
trajectories, selecting the most likely one. Our method simplifies the model
and enhances prediction accuracy, as shown in the results section.

In the present study, the scope of this research is to enhance the precise
prediction of the ego vehicle trajectory and contrast it with existing
benchmarks. The primary achievements of this research include:

  1. 1.

We developed a model to predict ego vehicle trajectory using HD map data. The
detailed environmental features and vehicle states are processed to generate
multiple possible trajectories.

  2. 2.

We compared our model with four leading baseline models for trajectory
prediction using the nuScenes dataset [Caesar et al., 2020]. Our evaluation
showed how our model performs against established methods, demonstrating its
effectiveness in predicting vehicle paths.

##  2 Research Background

Research into predicting the future movement of vehicles has been ongoing for
many years. Initial studies [Kaempchen et al., 2004, Lytrivis et al., 2008]
utilized physics-based kinematic models to estimate future positions,
demonstrating good accuracy in short-term trajectory [Sharma et al., 2023].
However, these models struggle to accurately predict long-term vehicle
behaviour, which is heavily influenced by external factors such as weather
conditions, traffic conditions, and the intentions of the drivers such as lane
changes, navigation goals [Fang et al., 2024, Cong et al., 2023]. More recent
investigations have turned to deep learning techniques, specifically recurrent
networks, to enhance prediction accuracy. These methods incorporate historical
data about the actors involved and contextual information from neighbouring
entities, offering a more nuanced approach to vehicle movement forecasting.

Multimodal prediction is a technique used to predict a set of possible paths
for each object involved, taking into account its unpredictable behaviour,
such as changes in speed and direction. To address the difficulty of
predicting multiple outcomes, deep learning techniques are often used. These
include generative adversarial networks [Goodfellow et al., 2014], and Graph
neural networks [Sharma et al., 2024b]. Each of these systems uses unique
approaches to solve multimodal trajectory problems. Recent studies by [Cheng
et al., 2021, Yuan et al., 2021, Lee et al., 2022] and [Chen et al., 2022] are
based on the concept of using a conditional variational autoencoder to predict
multiple vehicle trajectories efficiently. Despite their advantages, these
sampling-based methods lack an explicit way to calculate the probability of
each predicted path during the random selection stage. Additionally, with the
remarkable progress in computer vision, diffusion models have begun to be used
to understand the behaviour of road users and generate various vehicle
trajectory predictions. However, these models usually require a lot of time
due to the use of sequential sampling procedures. Inspired by these previous
methods, we designed our model, MapsTP, which estimates precise trajectory
predictions of the ego vehicle using HD map images and a spatio-temporal
probabilistic network.

##  3 Our Approach

In this section, we explain how our proposed architecture works and detail
each component involved. Our methodology integrates HD maps, feature
extraction, and a spatio-temporal probabilistic network to predict vehicle
trajectories accurately.

Figure 2: Our proposed architecture: HD Map Integration for Vehicle Trajectory
Planning. The process begins with the feature extractor analyzing these inputs
to create image features. These features are then analyzed by a probabilistic
network, which evaluates multiple potential trajectories and assigns
probabilities to each.

###  3.1 HD Maps

HD maps are essential for providing detailed and accurate information about
the environment surrounding the vehicle. These maps include precise lane
markings, road boundaries, traffic signs, and other critical features that are
necessary for safe navigation. By utilizing HD maps, the system can have a
comprehensive understanding of the road structure and layout, which is crucial
for accurate trajectory prediction.

###  3.2 Feature Extractor

The feature extractor (ResNet-50) block is responsible for processing HD map
data and sensor inputs to generate meaningful features for the prediction
model. The sensor inputs typically include yaw rate, speed, and acceleration,
providing real-time information about the vehicle’s state. The feature
extractor converts raw data from the IMU sensor into discrete segments
corresponding to vehicle movement over time, highlighting important
characteristics such as lane positions, intersections, and road geometry.
These extracted features are then fed into the next stage of the system.

###  3.3 Spatio-Temporal Probabilistic Network (STPN)

The STPN is designed to predict possible trajectories based on the extracted
features. This network considers the spatial relationships (such as road
layout and agents) and temporal dynamics (such as vehicle speed and
acceleration) to generate a set of probable trajectories
$P_{1},P_{2},\ldots,P_{n}$. Each individual trajectory $P_{i}$ is assigned a
probability score that indicates the likelihood of it being the correct
prediction.

The probabilistic network can be represented by a function $g$ that maps the
feature vectors $F$ to a set of possible trajectories with associated
probabilities:

$\\{(P_{i},\text{prob}(P_{i}))\\}=g(F)\quad\text{for}\quad 1\leq i\leq N$

where $\text{prob}(P_{i})$ is the probability of the $i$-th trajectory. we
typically denote the range of indices over which $i$ varies. If there are $N$
possible trajectories, the index $i$ ranges from 1 to $N$

###  3.4 Maximum Probability and Output Trajectory

Once the trajectories are generated, the system identifies the trajectory with
the maximum probability. This trajectory represents the most likely path the
vehicle will follow. The final step involves projecting this trajectory back
onto the HD map, resulting in the output trajectory that guides the vehicle’s
movement.

The output trajectory ensures that the vehicle follows a safe and efficient
path, taking into account the detailed road information from the HD maps and
the real-time state inputs.

##  4 Implementation Details

We implemented our method using ResNet-50 [He et al., 2016] with pre-trained
ImageNet [Russakovsky et al., 2015] weights as the backbone. High-level
features are extracted from the conv5 layer and globally pooled, then combined
with an agent state vector that includes velocity, acceleration, and yaw rate
as described in [Cui et al., 2019]. This model predicts trajectories within a
6-second horizon. Training parameters include a fixed learning rate of $1e-4$,
a batch size of $32$, and $50$ epochs. This setup effectively integrates
visual and dynamic information to achieve reliable trajectory prediction.

##  5 Experimental setup and Evaluations

In this section, we discuss the dataset utilized in our research and delve
into the specifics of the evaluation metrics employed to assess the
performance of our baseline models.

###  5.1 Nuscenes Dataset

To evaluate our method, we considered nuScenes dataset [Caesar et al., 2020],
which comprises $1000$ video sequences captured in Boston and Singapore. Each
video is 20 seconds in duration and contains 40 frames, resulting in a total
of 40,000 samples. The dataset is divided into training, validation, and
testing sets, with each set containing 700, 150, and 150 scenes, respectively.

###  5.2 Evaluation Metrics

We calculate evaluation metrics for our model by comparing it with state-of-
the-art models, using a comprehensive set of performance metrics to evaluate
its effectiveness and robustness, as in previous studies [Phan-Minh et al.,
2020, Chai et al., 2019]

#### MinADEk

: We compute the minimum average and final displacement errors across $k$
predicted trajectories for ego vehicle trajectory prediction.

$\mathbf{MinADE_{k}}=\min_{i\in\\{1...K\\}}\frac{1}{T_{f}}\sum_{t=1}^{T_{f}}\left\|y_{t}^{\text{gt}}-y_{t}^{(i)}\right\|_{2}$
(1)

where $y^{gt}$ represents the ground truth position of the object at the final
time step T, and $y_{t}^{(i)}$ represents the predicted position of the object
at the final time step T for the $i_{th}$ trajectory in the set of k
trajectories.

#### MinFDEk

: We evaluate the minimum final displacement error across $k$ predicted
trajectories for ego vehicle trajectory prediction.

$\mathbf{MinFDE_{k}}=\min_{i\in\\{1...K\\}}\left\|y_{t}^{\text{gt}}-y_{t}^{(i)}\right\|_{2}$
(2)

#### MissRatek,d

: Similar to [Messaoud et al., 2020], to calculate the miss rate, we consider
a given distance $d$ and the $k$ most likely predictions generated by the STPN
model. If the minimum distance between these predictions and the ground truth
is greater than d, we count it as a miss.

$\mathbf{MissRate_{k,\textit{d}}}=\begin{cases}1&\text{if}\ \min_{\hat{y}\in
P_{k}}\left(\max_{t=P_{t}}^{t=F_{t}}\left\|y_{t}^{\text{gt}}-y_{t}^{(i)}\right\|\right)\geq
d\\\ 0&\text{otherwise}\end{cases}$ (3)

Where ${F_{t}}$ donated the true future time step, whereas ${P_{t}}$
represented the predicted future time step.

##  6 Results

In this section, we present our analysis, which includes both quantitative and
qualitative components. Our quantitative analysis provides a statistical
overview of the data, highlighting key trends and patterns, while our
qualitative analysis offers in-depth insights and a contextual understanding
of the results.

###  6.1 Quantitative Analysis

We carry out a thorough experimental analysis, comparing our model against
four benchmark models:
[Phan-Minh et al., 2020], [Cui et al., 2019], [Chai et al., 2019], and
[Salzmann et al., 2021]. These benchmarks represent the state-of-the-art in
trajectory prediction according to recent proposed models. The purpose of this
comparison is to determine how well our model performs in predicting
trajectories compared to these established models. We seek to establish if our
model outperforms or aligns with the capabilities of current benchmarks.
Through this evaluation, we aim to identify the model’s strengths and
weaknesses, which helps us to identify areas for potential improvement. To
assess our model’s performance on the nuScenes dataset, we examine the
generated trajectory sequence $[y_{1},y_{2},y_{3},\ldots,y_{n}]$.

We evaluated the performance of the model on the dataset for different values
of $K$, where $K$ is set to 5, 10, and 15. To compute MinADEk, MinFDEk and
MissRatek,d we use equation 1, 2 and 3. During the training phase, our goal
was to minimize the MinADEk on the training set, as outlined in Table 1.
Essentially, we sought to lessen the discrepancy between the predicted and
actual trajectories by minimizing the smallest difference between them across
each of the $K$ time intervals. This strategy enabled us to enhance the
precision of our model’s predictions and confirm its efficacy on the nuScenes
dataset.

Method | MinADE5 $\downarrow$ | MinADE${}_{10}\downarrow$ | MinADE${}_{15}\downarrow$ | MinFDE5 $\downarrow$ | MinFDE10 $\downarrow$ | MinFDE15 $\downarrow$ | MissRate5,2 $\downarrow$ | MissRate10,2 $\downarrow$
---|---|---|---|---|---|---|---|---
CoverNet [Phan-Minh et al., 2020] | 2.62 | 1.92 | 1.63 | 11.36 | - | - | 0.76 | 0.64
Trajectron++ [Salzmann et al., 2021] | 1.88 | 1.51 | - | - | - | - | 0.70 | 0.64
MTP [Cui et al., 2019] | 2.22 | 1.74 | 1.55 | 4.83 | 3.54 | 3.05 | 0.74 | 0.67
MultiPath [Chai et al., 2019] | 1.78 | 1.55 | 1.52 | 3.62 | 2.93 | 2.89 | 0.78 | 0.76
MHA-JAM [Messaoud et al., 2020] | 1.85 | 1.24 | 1.03 | 3.72 | 2.23 | 1.67 | 0.60 | 0.46
GoalNet [Zhang et al., 2020] | 1.80 | 1.64 | 1.52 | 4.65 | 3.83 | - | 0.58 | 0.45
MapsTP (Ours) | 1.30 | 1.12 | 1.11 | 3.58 | 2.19 | 1.55 | 0.51 | 0.42

Table 1: Comparison of Methods on nuScenes Dataset [Caesar et al., 2020]:
Minimum Average Prediction Error (MinADE) and Final Displacement Error
(MinFDE) Analysis over a 6-second prediction horizon

###  6.2 Qualitative Analysis

As illustrated in Figure 3, on the left side, we utilize HD map or raster map
data from Nuscene. The HD map provides detailed and precise information
regarding the environment surrounding the vehicle. These map information
include accurate lane markings, road boundaries, traffic signs, and other
objects present in the scene. In the middle, the multi-model trajectory
prediction is depicted in red. This visualization showcases multiple possible
trajectories, each representing a potential path that a vehicle might take,
considering various factors such as speed and direction. It provides valuable
insights into the diverse range of paths that vehicles could follow in
different scenarios. Note that, our model’s trajectory predictions are well-
represented, as evidenced by the ground truth depicted on the right-hand side
in green. This alignment with ground truth demonstrates the efficacy of our
model in accurately forecasting vehicle trajectories. In Figure 3, we consider
the highest probability to predict the trajectory, but we also take into
account other probabilities for different scenarios.

Figure 3: Qualitative Analysis: HD Map Integration for Vehicle Trajectory
Prediction. The process begins with the feature extractor analyzing these
inputs to create image features. These features are then analyzed by a
probabilistic network, which evaluates multiple potential trajectories and
assigns probabilities to each. On the left, the HD image is displayed; the
middle section shows the multimodal trajectory prediction in red; and on the
right, the ground truth trajectory is shown in green.

##  7 Conclusion

In this paper, we present a straightforward solution for MapsTP by utilizing
them alongside state variables as inputs. We employ a basic CNN architecture
like ResNet-50 to extract features from the HD map. Subsequently, we employ an
STPN to compute potential trajectories. Our method shows promising results for
accurate and efficient trajectory prediction, outperforming current state-of-
the-art techniques. The multi-model trajectory prediction allows for the
calculation of multiple possible paths in a scene, aiding in better and more
precise decision-making. In future research, we aim to enhance trajectory
prediction for vehicles by integrating multiple sensors such as Lidar and
radar. Additionally, we plan to incorporate other datasets like Argoverse to
further improve the robustness and generalization of our approach.

## Acknowledgments

This publication has emanated from research supported in part by a grant from
Science Foundation Ireland under Grant number 18/CRT/6049. For the purpose of
Open Access, the author has applied a CC BY public copyright license to any
Author Accepted Manuscript version arising from this submission.

## References

  * [Bao et al., 2023] Bao, Z., Hossain, S., Lang, H., and Lin, X. (2023).  A review of high-definition map creation methods for autonomous driving.  Engineering Applications of Artificial Intelligence, 122:106125.
  * [Caesar et al., 2020] Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E., Xu, Q., Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. (2020).  nuscenes: A multimodal dataset for autonomous driving.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631.
  * [Chai et al., 2019] Chai, Y., Sapp, B., Bansal, M., and Anguelov, D. (2019).  Multipath: Multiple probabilistic anchor trajectory hypotheses for behavior prediction.
  * [Chehri and Mouftah, 2019] Chehri, A. and Mouftah, H. T. (2019).  Autonomous vehicles in the sustainable cities, the beginning of a green adventure.  Sustainable Cities and Society, 51:101751.
  * [Chen et al., 2022] Chen, Y., Ivanovic, B., and Pavone, M. (2022).  Scept: Scene-consistent, policy-based trajectory predictions for planning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17103–17112.
  * [Cheng et al., 2021] Cheng, H., Liao, W., Tang, X., Yang, M. Y., Sester, M., and Rosenhahn, B. (2021).  Exploring dynamic context for multi-path trajectory prediction.  In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 12795–12801.
  * [Cong et al., 2023] Cong, P., Xiao, Y., Wan, X., Deng, M., Li, J., and Zhang, X. (2023).  Dacr-amtp: adaptive multi-modal vehicle trajectory prediction for dynamic drivable areas based on collision risk.  IEEE Transactions on Intelligent Vehicles.
  * [Cui et al., 2019] Cui, H., Radosavljevic, V., Chou, F.-C., Lin, T.-H., Nguyen, T., Huang, T.-K., Schneider, J., and Djuric, N. (2019).  Multimodal trajectory predictions for autonomous driving using deep convolutional networks.
  * [Elghazaly et al., 2023] Elghazaly, G., Frank, R., Harvey, S., and Safko, S. (2023).  High-definition maps: Comprehensive survey, challenges and future perspectives.  IEEE Open Journal of Intelligent Transportation Systems.
  * [Fang et al., 2024] Fang, J., Wang, F., Xue, J., and Chua, T.-S. (2024).  Behavioral intention prediction in driving scenes: A survey.  IEEE Transactions on Intelligent Transportation Systems.
  * [Goodfellow et al., 2014] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014).  Generative adversarial nets.  Advances in neural information processing systems, 27.
  * [Guo et al., 2023] Guo, H., Meng, Q., Zhao, X., Liu, J., Cao, D., and Chen, H. (2023).  Map-enhanced generative adversarial trajectory prediction method for automated vehicles.  Information Sciences, 622:1033–1049.
  * [Hancock et al., 2019] Hancock, P. A., Nourbakhsh, I., and Stewart, J. (2019).  On the future of transportation in an era of automated and autonomous vehicles.  Proceedings of the National Academy of Sciences, 116(16):7684–7691.
  * [He et al., 2016] He, K., Zhang, X., Ren, S., and Sun, J. (2016).  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778.
  * [Huang et al., 2023] Huang, Z., Liu, H., Wu, J., and Lv, C. (2023).  Differentiable integrated motion prediction and planning with learnable cost function for autonomous driving.  IEEE transactions on neural networks and learning systems.
  * [Kaempchen et al., 2004] Kaempchen, N., Weiss, K., Schaefer, M., and Dietmayer, K. C. (2004).  Imm object tracking for high dynamic driving maneuvers.  In IEEE Intelligent Vehicles Symposium, 2004, pages 825–830. IEEE.
  * [Lee et al., 2022] Lee, M., Sohn, S. S., Moon, S., Yoon, S., Kapadia, M., and Pavlovic, V. (2022).  Muse-vae: Multi-scale vae for environment-aware long term trajectory prediction.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2221–2230.
  * [Lytrivis et al., 2008] Lytrivis, P., Thomaidis, G., and Amditis, A. (2008).  Cooperative path prediction in vehicular environments.  In 2008 11th International IEEE Conference on Intelligent Transportation Systems, pages 803–808. IEEE.
  * [Messaoud et al., 2020] Messaoud, K., Deo, N., Trivedi, M. M., and Nashashibi, F. (2020).  Trajectory prediction for autonomous driving based on multi-head attention with joint agent-map representation.
  * [Phan-Minh et al., 2020] Phan-Minh, T., Grigore, E. C., Boulton, F. A., Beijbom, O., and Wolff, E. M. (2020).  Covernet: Multimodal behavior prediction using trajectory sets.
  * [Russakovsky et al., 2015] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015).  Imagenet large scale visual recognition challenge.  International journal of computer vision, 115:211–252.
  * [Salzmann et al., 2021] Salzmann, T., Ivanovic, B., Chakravarty, P., and Pavone, M. (2021).  Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data.
  * [Sharma et al., 2024a] Sharma, S., Das, A., Sistu, G., Halton, M., and Eising, C. (2024a).  Bevseg2tp: Surround view camera bird’s-eye-view based joint vehicle segmentation and ego vehicle trajectory prediction.  In Proceedings of the 19th International Conference on Computer Vision Theory and Applications (VISAPP), Rome, Italy.
  * [Sharma et al., 2024b] Sharma, S., Singh, A., Sistu, G., Halton, M., and Eising, C. (2024b).  Optimizing ego vehicle trajectory prediction: The graph enhancement approach.  Electronic Imaging, 36:1–6.
  * [Sharma et al., 2023] Sharma, S., Sistu, G., Yahiaoui, L., Das, A., Halton, M., and Eising, C. (2023).  Navigating uncertainty: The role of short-term trajectory prediction in autonomous vehicle safety.  In Proceedings of the Irish Machine Vision and Image Processing Conference.
  * [Yuan et al., 2021] Yuan, Y., Weng, X., Ou, Y., and Kitani, K. (2021).  Agentformer: Agent-aware transformers for socio-temporal multi-agent forecasting.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV).
  * [Zhang et al., 2020] Zhang, L., Su, P.-H., Hoang, J., Haynes, G. C., and Marchetti-Bowick, M. (2020).  Map-adaptive goal-based trajectory prediction.
  * [Zhao et al., 2017] Zhao, Z., Zhou, L., Zhu, Q., Luo, Y., and Li, K. (2017).  A review of essential technologies for collision avoidance assistance systems.  Advances in Mechanical Engineering, 9(10):1687814017725246.


# Iterative, Deep, and Unsupervised
Synthetic Aperture Sonar Image Segmentation

Yung-Chen Sun1, Isaac D. Gerg12, Vishal Monga1  1Department of Electrical
Engineering, Pennsylvania State University, University Park, PA
2Applied Research Laboratory, Pennsylvania State University, State College, PA

###### Abstract

Deep learning has not been routinely employed for semantic segmentation of
seabed environment for synthetic aperture sonar (SAS) imagery due to the
implicit need of abundant training data such methods necessitate. Abundant
training data, specifically pixel-level labels for all images, is usually not
available for SAS imagery due to the complex logistics (e.g., diver survey,
chase boat, precision position information) needed for obtaining accurate
ground-truth. Many hand-crafted feature based algorithms have been proposed to
segment SAS in an unsupervised fashion. However, there is still room for
improvement as the feature extraction step of these methods is fixed. In this
work, we present a new iterative unsupervised algorithm for learning deep
features for SAS image segmentation. Our proposed algorithm alternates between
clustering superpixels and updating the parameters of a convolutional neural
network (CNN) so that the feature extraction for image segmentation can be
optimized. We demonstrate the efficacy of our method on a realistic benchmark
dataset. Our results show that the performance of our proposed method is
considerably better than current state-of-the-art methods in SAS image
segmentation.

###### Index Terms:

Seabed texture segmentation, superpixel segmentation, deep learning,
unsupervised learning, synthetic aperture sonar (SAS).

##  I Introduction

Recently, deep learning frameworks applied for image segmentation have shown
promising results [1, 2]. To segment distinct seabed textures in sonar images,
a naive intuition is that we can use a similar network structure for seabed
segmentation of SAS images because they have largely composted low-level
textures. However, training a convolutional neural network (CNN) requires a
considerable amount of labeled data to obtain good results. For SAS image
segmentation, this means acquiring pixel-level annotations which is a costly
process, and rarely do we have complete ground-truth from a diver survey.
Consequently, humans often label regions which are easily recognized while
forgoing areas difficult to distinguish such as region boundaries or blended
regions. As a result, few pixel-level labeled datasets are available for
training.

|
---|---
Figure 1: Sample SAS images containing various seabed types including large
sand ripples and flat sandy seafloor.

Unsupervised image classification [3] has shown that embedding deep learning
steps into a clustering algorithm is a promising direction. The essence of
image-level clustering is to see every image as a region that only contains
one high-level semantic texture, but the regions’ boundaries are inaccurate.
We apply this idea to the image segmentation task. Although we are going to do
pixel-level segmentation, from our observations regions containing a single
texture will produce similar pixel predictions. If we can find accurate
boundaries to differentiate the regions containing a single texture, then the
image segmentation process can be transferred into the region-level
classification problem.

We devise a novel unsupervised segmentation method that combines deep learning
and an iterative clustering technique producing an algorithm suitable for
unsupervised seabed segmentation of SAS images. The basic idea is to use
superpixels generated by CNN pseudo-labels to split one sonar image into many
small regions. Each region is quantized into one superpixel feature that is
the average softmax value and then a clustering process is performed to group
these regions. Finally, we utilize the clustering results to update the CNN
parameters to encourage more accurate predictions of a segmentation network.

We evaluate our method, which we call iterative deep unsupervised segmentation
(IDUS), against several comparison methods on a contemporary real-world SAS
dataset. We show compelling performance of IDUS against existing state-of-the-
art methods for SAS image co-segmentation. Additionally, we show an extension
of our method which can obtain even better results when semi-labeled data is
present. Specifically, our work makes two main contributions:

  1. 1.

We devise a new iterative method to integrate the merits of best-known
unsupervised superpixel segmentation along with a deep feature extractor.

  2. 2.

Our approach does not depend on any specific network structure or superpixel
generation algorithm. The user can adjust the algorithm easily according to
different segmentation tasks.

##  II Technical Approach

---
Figure 2:  An iteration of IDUS: (1) Use SLIC [4] to generate the new
superpixel segments. (2) Quantize pixel features into superpixel features. (3)
Cluster superpixel features by k-means to update the class assignments in
memory. (4) Map superpixel labels to pseudo pixel labels. (5) Compute the loss
via the pseudo pixel labels and the network softmax outputs. (6) Update the
network via backpropagation.

A recent method called DeepCluster [3] demonstrates that the combination of
clustering techniques and CNNs show promise in unsupervised learning of deep
features on image classification dataset is possible. The benefit of
DeepCluster is that the training method is more generally and easily
transferred to other tasks. This benefit motivates us to develop our own work
on unsupervised image segmentation because the method not only will not be
limited to the sonar images, but it can also be used in other segmentation
tasks.

Notation. Given a set
$\mathbf{X}=\left\\{\mathbf{x_{1}},...,\mathbf{x_{N}}\right\\}$ of $N$ SAS
images, $s_{n}^{k}$ is the $k$-th superpixel of the $n$-th image and
$r_{n}^{k}$ are the pseudo-label of this superpixel. We use
$h_{\theta}(\cdot)$ to denote the softmax output of a segmentation network
(i.e. deep network), where $\theta$ is the parameters we need to optimize. The
output has same size with the input image. Given an input image
$\mathbf{x_{n}}$, the network maps the image into a feature map
$\mathbf{f_{n}}=h_{\theta}(\mathbf{x_{n}})$ in which $f_{n}^{i}$ represent the
feature of $i$-th pixel $x_{n}^{i}$ in this image.

Superpixel Pooling. To quantize the pixel-level features into region-level
features
$\mathbf{z_{n}}=Q_{pool}(\mathbf{x_{n}},\mathbf{f_{n}},\mathbf{s_{n}})$, we
compute the feature’s mean from all the pixels making up the superpixel
$z_{n}^{k}$ by the following equation:

$z_{n}^{k}=\frac{1}{D}\sum_{i=1}^{D}f_{n}^{i}I(x_{n}^{i}\in s_{n}^{k}),$ (1)

where $D$ denotes the number of pixels in the input image. $I(x_{n}^{i}\in
s_{k}^{i})$ is an indicator function that is 1 if $x_{n}^{i}$ belongs to the
superpixel $s_{n}^{k}$. The superpixel quantization makes sure that we can
collect regional features of sufficient accuracy for the later clustering use.

Superpixel Mapping. Suppose we have already known a superpixel label
$r_{n}^{k}$, then every pixel label $y_{n}^{i}$ in this superpixel should be
the same. We use the following formula to describe this mapping:

$y_{n}^{i}=\sum_{k=1}^{K_{n}}r_{n}^{k}I(x_{n}^{i}\in s_{n}^{k})$ (2)

where $K_{n}$ denotes the number of superpixels in $n$th image. We use
$\mathbf{y_{n}}=Q_{map}(\mathbf{x_{n}},\mathbf{r_{n}},\mathbf{s_{n}})$ to
denote this process which we call superpixel mapping. It eliminates some
classification errors and produces more robust class assignments for pixels.

The Simplest Version of IDUS. Figure 2 shows an iteration of the simplest
version of IDUS after initialization. Here, we assume that the minibatch only
contains one image. The superpixel segments memory stores the boundary of
every superpixel and the superpixel labels memory stores the superpixel
labels. We see the softmax vectors of network outputs as a kind of feature
representation, due to it has the same size of the inputs and contains the
distance information between a region texture and cluster centroids. In an
iteration, we first use superpixel segments to quantize the pixel features
into superpixel features (Eq. (1)) and use $k$-means to update the superpixel
labels. Next, we map the superpixel labels to pixel labels (Eq. (2)) and use
the pseudo-class assignments to update the network via back-propagation. As
shown in Figure 2, IDUS alternates between generating pseudo ground-truth
labels and updating the network parameters.

The Complete Version of IDUS Given a set of training images
$\mathbf{X_{T}}=\left\\{\mathbf{x_{n}}\right\\}_{n=1}^{N_{T}}$, we update the
model’s parameters $\theta$ by solving the following problem,

$\min_{\theta}\frac{1}{N_{T}}\sum_{n=1}^{N_{T}}\ell(h_{\theta}(\mathbf{x_{n}}),Q_{map}(\mathbf{x_{n}},\mathbf{r_{n}},\mathbf{s_{n}})),$
(3)

where $\ell$ is a loss function and $\mathbf{r_{n}}$, $\mathbf{s_{n}}$ will be
initialized before performing the algorithm. After training the model for
$U_{E}$ epochs, we extract the feature maps
$\left\\{\mathbf{f_{n}}\right\\}_{n=1}^{N_{T}}$ of all training images based
on the current model’s parameters $\theta_{i}$ where $i$ is the number of
iteration. Then, we pool the feature maps to regional features
$\left\\{\mathbf{z_{n}}\right\\}_{n=1}^{N_{T}}$ and cluster them into $M$
groups based on the optimization goal

$\min_{C\in\mathbb{R}^{M\times
d}}\sum_{m=1}^{M}\sum_{n=1}^{N}\sum_{k=1}^{K_{n}}\min_{r_{n}^{k}\in[1,M]}\|z_{n}^{k}-c_{m}\|I(r_{n}^{k}==m),$
(4)

where $c_{m}$ is the centroid feature of class $m$ and $d$ is the length of
feature dimension. With this clustering criterion, the regional features will
be clustered and used to produced new labels
$\left\\{\mathbf{r_{n}}\right\\}_{n=1}^{N_{T}}$; and then, they will be
assigned to the corresponding superpixels.

The overall idea of IDUS can be summarized in Algorithm 1. We update the
superpixel boundaries every $U_{S}$ epochs and the algorithm terminates after
a fixed number of iterations.

Algorithm 1 Iterative Deep Unsupervised Segmentation

1: Initialize superpixel segments, labels
$\left\\{\mathbf{s_{n}},\mathbf{r_{n}}\right\\}_{n=1}^{N_{T}}$ of $N_{T}$
training images. And map the regional labels to pseudo pixel labels
$\left\\{\mathbf{y_{n}}\right\\}_{n=1}^{N_{T}}$ by Eq 2.

2: Assign intervals $U_{E}$, $U_{S}$ for updating the pseudo-ground-truth and
superpixel boundaries.

3: repeat

4: Given training data set
$\left\\{\mathbf{x_{n}},\mathbf{y_{n}}\right\\}_{n=1}^{N_{T}}$, update the
model’s parameters $\theta$ based on the problem Eq (3).

5: if every $U_{S}$ epochs then

6: Update superpixel boundaries $\left\\{\mathbf{s_{n}}\right\\}_{n=1}^{N}$
based on the model of current iteration.

7: end if

8: if every $U_{E}$ epochs then

9: Use the model parameters of current iteration $\theta_{i}$ to generate
pixel-level features $\\{\mathbf{f_{n}}\\}_{n=1}^{N_{T}}$.

10: Pool the pixel-level features into regional features by Eq (1).

11: Cluster the regional features based on criterion Eq (4) and reassign new
pseudo labels $\left\\{\mathbf{r_{n}}\right\\}_{n=1}^{N_{T}}$ to each
superpixel.

12: end if

13: until reach the maximum iteration.

14: Output the model’s parameters $\theta$.

Layer Name | Layer Function | Dim. | # Filters | Input
---|---|---|---|---
up1 | Upsampling | 2$\times$2 | N/A | res4
merge1 | Concatenate | N/A | N/A | up1, res3
conv1a | Conv+BN+ReLU | 3$\times$3 | 256 | merge1
conv1b | Conv+BN+ReLU | 3$\times$3 | 256 | conv1a
up2 | Upsampling | 2$\times$2 | N/A | conv1b
merge2 | Concatenate | N/A | N/A | up2, res2
conv2a | Conv+BN+ReLU | 3$\times$3 | 128 | merge2
conv2b | Conv+BN+ReLU | 3$\times$3 | 128 | conv2a
up3 | Upsampling | 2$\times$2 | N/A | conv2b
merge3 | Concatenate | N/A | N/A | up3, res1
conv3a | Conv+BN+ReLU | 3$\times$3 | 64 | merge3
conv3b | Conv+BN+ReLU | 3$\times$3 | 64 | conv3a
up4 | Upsampling | 2$\times$2 | N/A | conv3b
merge4 | Concatenate | N/A | N/A | up4, res_conv
conv4a | Conv+BN+ReLU | 3$\times$3 | 32 | merge4
conv4b | Conv+BN+ReLU | 3$\times$3 | 32 | conv4a
up5 | Upsampling | 2$\times$2 | N/A | conv4b
conv5a | Conv+BN+ReLU | 3$\times$3 | 16 | up5
conv5b | Conv+BN+ReLU | 3$\times$3 | 16 | conv5a
seg_head | Conv+Softmax | 3$\times$3 | 7 | conv5b
TABLE I: Description of decoder and segmentation head of the U-Net used in
IDUS. This decoder is preceded by a pre-trained ResNet-18 network.
“Conv+BN+ReLU” is 2D convolutional followed by Batch Normalization followed by
rectified linear unit (ReLU) activation and “Softmax” denotes the softmax
function. We use layers res_conv, res1, res2, res3, and res4 to denote the
output of ResNet-18’s first convolutional layer and building blocks,
respectively. Layer conv5b is the decoder output and layer seg_head is the
segmentation head output.

Implementation Details The backbone model used by IDUS is U-Net[2] equipped
with ResNet-18 encoder that pre-trained on ImageNet [5]. TABLE I shows the
decoder and segmentation head (pixel-level linear classifier) of U-Net. For
the initialization, we first concatenate the feature maps of the output of the
first and third convolutional blocks in ResNet-18 of every image; and then, we
use the idea of texton [6] to initialize the pseudo labels and a two-step
texton selection [7] to reduce the computation time. We train the IDUS with
mini-batch size fifteen on an NVIDIA Titan X GPU (12GB) with the PyTorch
package [8]. The loss function we use is the mean of dice loss [9] and cross-
entropy loss. We use weights $w_{m}^{ept}=1/r_{m}$ and
$w_{m}^{dice}=1/\sqrt{r_{m}}$, where $r_{m}$ was the proportion of $m$th class
label in training samples, to balance the class weights of two losses,
respectively. Batch normalization layers [10] are added after each
convolutional layer (except the segmentation head) to accelerate the loss
convergence. The model’s parameters are initialed by a uniform distribution
following the scaling of [11] and optimized by Adam [12] algorithm with weight
decay value $10^{-9}$. In every iteration, the learning rate of the model is
initialized at $10^{-4}$ and decreased by $0.1$ every fifty epochs. The
updating interval of superpixel labels and boundaries is 200 and we do this
five times throughout training. Consequently, we train for 1000 epochs where
every 200 epochs we update the superpixel labels and boundaries.

##  III Experimental Results

###  III-A Experimental Setup

The dataset we use to test the performance of IDUS is composed of 113 images
collected from a high-frequency SAS system which are semi-labeled (i.e. not
every pixel is labeled in every image). The images contain seven distinct
labeled seabed areas: shadow (SH), dark sand (DS), bright sand (BS), seagrass
(SG), large sand ripple (SR), rock (RK), and small sand ripple (SR). The
images were originally $1001\times 1001$ pixels in size which we downsample to
$512\times 512$ pixels for computational efficiency.

Comparison Methods. To show the effectiveness of our method, we compare with
two recent methods [13, 14] used in SAS seabed environment image segmentation.
Following, we give the implementation details we use in generating the
comparison methods as no source code is publicly available to evaluate. We
make a best effort attempt to reproduce the methods as given in their
respective sources.

Lianantonakis, et al. (2007) [13]. This method uses Haralick features [15]
derived from the gray-level co-occurrence matrix (GLCM) and couples this with
active contours to arrive at a binary class mapping. We extend this work to
multiple classes by simply using the same feature descriptors as the original
work but apply $k$-means++ to cluster; a similar replication approach is used
in [16]. We ran $k$-means++ with 100 random initializations and selected the
run that produces the minimum within cluster sum of squares error in a manner
consistent with [17].

Zare, et al. (2017) [14]. In this work, the feature sets are produced by Sobel
edge descriptors (Sobel) [18], histograms of oriented gradients (HOG) [19],
and local binary pattern (LBP) features [20]. For each feature descriptor, we
use the same sliding window strategy of Lianantonakis, et al.[13] to derive a
feature vector for each pixel.

For the comparison methods, the co-segmentation strategy contains four main
steps:

  1. 1.

Feature extraction to extract a set of feature maps from the input sonar
images.

  2. 2.

Performing SLIC [4] on the extracted features to generate superpixel features
by Eq (1).

  3. 3.

Unsupervise cluster all the superpixels and provide each superpixel a cluster
assignment.

  4. 4.

Pixels inside a superpixel are assigned the same cluster by Eq (2) to get
pixel-level co-segmentation.

For the co-segmentation strategy of IDUS, we train a segmentation network by
IDUS on the entire dataset first, and then directly use the network outputs
computed from the input sonar images as the final co-segmentation results.

###  III-B Results

---
(a) Lianantonakis, et al. (2007) [17] 2009
(b) Zare, et al. (2017) [14] 2017
(c) IDUS
Figure 3: Confusion matrices of IDUS along with comparisons in the first
experiment. Higher values along the diagonal indicate better performance.

We use the confusion matrix as a criterion to evaluate the co-segmentation
performance of IDUS and the comparison methods on the entire dataset (113
images). A confusion matrix shows the proportion of an assigned class
predicted by the algorithm for a given ground-truth class. Ideally, the
ground-truth class and the predicted class overlap entirely giving a
proportion of one. However, in practice, the predicted class by co-
segmentation usually do not perfectly match with the ground-truth class
resulting in a proportion less than one.

As a result of performing unsupervised segmentation, class assignments by the
algorithm must be mapped to ground-truth classes. We perform this assignment
by initially using a random disjoint assignment among the possible classes and
then compute the confusion matrix. Next, we re-sort the columns of the matrix
in such a way so that the sum of diagonal elements in the new confusion matrix
is maximized. For the co-segmentation of the comparison methods, we set the
clusters number is the same as the number of ground-truth classes (seven
class). For IDUS, we set the dimension of the network softmax output as seven.
Therefore, we use a confusion matrix with size $7\times 7$ as the criterion
for evaluating different methods.

Figure 3 shows the confusion matrices of IDUS and the comparison methods. As
shown in the figure, IDUS results in superior co-segmentation results (Figure
3c) over the comparison methods (Figure 3a and b). Upon examining the
confusion matrices, we notice some comparison methods do not provide enough
discrimination ability to differentiate certain classes well. For example,
Shadow (SH) and Dark Sand (DS) are clustered into the same class in
Lianantonakis, et al. [13] (Figure 3a). However, such misclassification
problems are diminished by IDUS (Figure 3c).

##  IV Conclusion

We propose an unsupervised method for SAS seabed image segmentation which
works by iteratively clustering on superpixels while training a deep model.
Our proposed method (iterative, deep, and unsupervised segmentation (IDUS))
does not require large amounts of labeled training data which are difficult to
obtain for SAS. We show that IDUS obtains state-of-the-art performance on a
real-world SAS dataset and show performance benefits by comparing against
existing methods.

In future work, we will extend IDUS to combine with supervised or semi-
supervised training data.

##  V Acknowledgements

This work was supported by the Office of Naval Research via grant
N00014-19-1-2513. The authors thank the Naval Surface Warfare Center - Panama
City Division for providing the data used in this work.

## References

  * [1] P. Liskowski and K. Krawiec, “Segmenting retinal blood vessels with deep neural networks,” _IEEE TMI_ , vol. 35, no. 11, pp. 2369–2380, 2016.
  * [2] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _MICCAI_ , 2015, pp. 234–241.
  * [3] M. Caron, P. Bojanowski, A. Joulin, and M. Douze, “Deep clustering for unsupervised learning of visual features,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 132–149.
  * [4] R. Achanta, A. Shaji, K. Smith, A. Lucchi, P. Fua, and S. Susstrunk, “SLIC superpixels compared to state-of-the-art superpixel methods,” _IEEE Trans. Pattern Anal. Mach. Intell._ , vol. 34, no. 11, p. 2274–2282, Nov. 2012\.
  * [5] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “ImageNet: A Large-Scale Hierarchical Image Database,” in _CVPR09_ , 2009.
  * [6] J. Malik, S. Belongie, T. Leung, and J. Shi, “Contour and texture analysis for image segmentation,” _International Journal of Computer Vision_ , vol. 43, no. 1, pp. 7–27, 2001.
  * [7] J. T. Cobb and A. Zare, “Multi-image texton selection for sonar image seabed co-segmentation,” in _Defense, Security, and Sensing_ , 2013.
  * [8] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [9] C. H. Sudre, W. Li, T. Vercauteren, S. Ourselin, and M. J. Cardoso, “Generalised dice overlap as a deep learning loss function for highly unbalanced segmentations,” in _Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support_. Springer, 2017, pp. 240–248.
  * [10] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _Proceedings of International Conference on Machine Learning_ , 2015, pp. 448–456.
  * [11] K. He, X. Zhang, S. Ren, and J. Sun, “Delving deep into rectifiers: Surpassing human-level performance on Imagenet classification,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2015, pp. 1026–1034.
  * [12] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [13] M. Lianantonakis and Y. R. Petillot, “Sidescan sonar segmentation using texture descriptors and active contours,” _IEEE Journal of Oceanic Engineering_ , vol. 32, no. 3, pp. 744–752, 2007.
  * [14] A. Zare, N. Young, D. Suen, T. Nabelek, A. Galusha, and J. Keller, “Possibilistic fuzzy local information C-Means for sonar image segmentation,” _2017 IEEE SSCI_ , pp. 1–8, 2017.
  * [15] R. M. Haralick, K. Shanmugam, and I. H. Dinstein, “Textural features for image classification,” _IEEE Transactions on Systems, Man, and Cybernetics_ , no. 6, pp. 610–621, 1973.
  * [16] J. T. Cobb and J. Principe, “Autocorrelation features for synthetic aperture sonar image seabed segmentation,” in _2011 IEEE International Conference on Systems, Man, and Cybernetics_. IEEE, 2011, pp. 3341–3346.
  * [17] D. P. Williams, “Unsupervised seabed segmentation of synthetic aperture sonar imagery via wavelet features and spectral clustering,” in _2009 16th IEEE International Conference on Image Processing (ICIP)_. IEEE, 2009, pp. 557–560.
  * [18] H. Frigui and P. Gader, “Detection and discrimination of land mines in ground-penetrating radar based on edge histogram descriptors and a possibilistic $k$-nearest neighbor classifier,” _IEEE Transactions on Fuzzy Systems_ , vol. 17, no. 1, pp. 185–199, 2009.
  * [19] N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in _IEEE Computer Vision and Pattern Recognition (CVPR’05)_ , vol. 1. IEEE, 2005, pp. 886–893.
  * [20] Z. Guo, L. Zhang, and D. Zhang, “A completed modeling of local binary pattern operator for texture classification,” _IEEE Transactions on Image Processing_ , vol. 19, no. 6, pp. 1657–1663, 2010.