**Application of Multi-Sensor Fusion Technology in Target Detection and Recognition**

Editors

**Jukka Heikkonen Fahimeh Farahnakian**

MDPI • Basel • Beijing • Wuhan • Barcelona • Belgrade • Manchester • Tokyo • Cluj • Tianjin

*Editors* Jukka Heikkonen Computing University of Turku Turku Finland

Fahimeh Farahnakian Computing University of Turku Turku Finland

*Editorial Office* MDPI St. Alban-Anlage 66 4052 Basel, Switzerland

This is a reprint of articles from the Special Issue published online in the open access journal *Remote Sensing* (ISSN 2072-4292) (available at: www.mdpi.com/journal/remotesensing/special issues/application multi-sensor fusion technology target detection recognition).

For citation purposes, cite each article independently as indicated on the article page online and as indicated below:

LastName, A.A.; LastName, B.B.; LastName, C.C. Article Title. *Journal Name* **Year**, *Volume Number*, Page Range.

**ISBN 978-3-0365-1352-2 (Hbk) ISBN 978-3-0365-1351-5 (PDF)**

© 2022 by the authors. Articles in this book are Open Access and distributed under the Creative Commons Attribution (CC BY) license, which allows users to download, copy and build upon published articles, as long as the author and publisher are properly credited, which ensures maximum dissemination and a wider impact of our publications.

The book as a whole is distributed by MDPI under the terms and conditions of the Creative Commons license CC BY-NC-ND.

## **Contents**

## **About the Editors**

### **Jukka Heikkonen**

Jukka Heikkonen is a full professor and head of the Algorithms and Computational Intelligence research Lab, University of Turku, Finland. His research focuses on data analytics, machine learning and autonomous systems. He has worked at top level research laboratories and Center of Excellences in Finland and international organizations (European Commission, Japan) and has led many international and national research projects. He has authored more than 150 peer-reviewed scientific articles. He has served as organizing/program committee member in numerous conferences, and acted as a guest editor in 5 special issues of scientific journals.

#### **Fahimeh Farahnakian**

Fahimeh Farahnakian is currently an adjunct professor (docent) in the Algorithms and Computational Intelligence research Lab, Department of Future Technologies, University of Turku, Finland. Her research interests include the theory and algorithms of machine learning, computer vision and data analysis methods, and their applications in various different fields. She have published +30 articles in journal and conference proceedings. She is a member of the IEEE and also served in program committees of numerous scientific conferences.

## **Preface to "Application of Multi-Sensor Fusion Technology in Target Detection and Recognition"**

Application of multi-sensor fusion technology has drawn a lot of industrial and academic interest in recent years. The multi-sensor fusion methods are widely used in many applications such as autonomous systems, remote sensing, video surveillance and military. These methods can obtain the complementary properties of targets by considering multiple sensors. On the other hand, they can achieve a detailed environment description and accurate detection of interest targets based on the information from different sensors.

This book collects novel developments in the field of multi-sensor, multi-source and multi-process information fusion. Articles are expected to emphasize one or more of the three facets: architectures, algorithms, and applications. Published papers dealing with fundamental theoretical analyses as well as those demonstrating their application to real-world problems.

> **Jukka Heikkonen, Fahimeh Farahnakian** *Editors*

## *Article* **Deep Learning Based Multi-Modal Fusion Architectures for Maritime Vessel Detection**

## **Fahimeh Farahnakian \*,† and Jukka Heikkonen †**

Department of Future Technologies, University of Turku, 20500 Turku, Finland<EMAIL_ADDRESS>

Received: 20 July 2020; Accepted: 2 August 2020; Published: 5 August 2020

**Abstract:** Object detection is a fundamental computer vision task for many real-world applications. In the maritime environment, this task is challenging due to varying light, view distances, weather conditions, and sea waves. In addition, light reflection, camera motion and illumination changes may cause to false detections. To address this challenge, we present three fusion architectures to fuse two imaging modalities: visible and infrared. These architectures can provide complementary information from two modalities in different levels: pixel-level, feature-level, and decision-level. They employed deep learning for performing fusion and detection. We investigate the performance of the proposed architectures conducting a real marine image dataset, which is captured by color and infrared cameras on-board a vessel in the Finnish archipelago. The cameras are employed for developing autonomous ships, and collect data in a range of operation and climatic conditions. Experiments show that feature-level fusion architecture outperforms the state-of-the-art other fusion level architectures.

**Keywords:** multi-sensor fusion; object detection; deep learning; convolutional neural networks; autonomous vehicles; marine environment

## **1. Introduction**

Object detection is a crucial problem for autonomous vehicles and has been studied for years to make it efficient and faster. A reliable autonomous driving system relies on accurate object detection for providing robust perception of the environment. In addition, the performance of subsequent tasks such as object classification and tracking depend strongly on the object detection. In marine environment, object detection is a challenging problem due to varying light, view distances, weather conditions, and dynamic sea nature. In addition, light reflection, camera motion and illumination changes may cause false detections [1].

Multi-sensor fusion technology is a promising solution for achieving accurate object detection by obtaining the complementary properties of objects based on multiple sensors. The multi-sensor fusion architectures are generally classified into three groups that are based on the level of data abstraction used for fusion [2]. (1) Early fusion, also called pixel-level fusion, combines raw data from the sensors before applying any information extraction strategies. (2) Middle fusion, also called feature-level fusion, fuses the extracted features from each raw sensor data and then performs detection on the fused data. (3) Late fusion, also called decision-level fusion, independently performs detection from each sensor and the outputs of each sensor are fused at the decision level for final detection.

Among the combination of sensor types, InfRared (IR) and visible (RGB) image fusion is superior in many aspects [3]. Firstly, image sensors are cheap when compared in other sensors, such as radar and LiDAR (Light Detection And Ranging). Secondly, collecting and annotating image data is much easier than LiDAR point clouds. Thirdly, IR and RGB images share complementary properties, thus producing robust and informative fused images. Finally, RGB images typically have high spatial resolution and considerable detail when compared to the images that obtained from other sensors. However, these images can be easily influenced by severe conditions, such as poor illumination, fog, and other effects of bad weather. Meanwhile, the thermal IR cameras capture relative temperature, which allows for distinguishing warm objects, like person from cold objects, like navigation buoy or the island. Moreover, IR cameras can improve navigation safety at night/day time and all-weather conditions by determining interest objects based on radiation difference [1–3].

Convolutional Neural Networks (CNNs) or ConvNet allowed for a significant improvement in the performance of computer vision tasks, such as object classification [4], detection [5,6], and segmentation [7]. Moreover, various fusion approaches have been employed CNN in autonomous vehicles [1,8,9]. While the majority of these approaches has focused on RGB images, some of them have also been directed using infrared images for object detection. We use CNN for addressing the object detection problem in marine environment to fill this gap and by the fact that CNN is a very powerful model for computer vision tasks.

In this work, we present three early, middle and late fusion CNN architectures to carry out vessel detection in marine environment. These architectures can fuse the images from the visible and thermal infrared cameras at the different levels of data abstraction. In addition, these architectures employed a deep CNN as a detector to generate bounding box proposals for interest vessels in marine environment. We did not take into consideration any semantic segmentation algorithms in this study. The CNN is trained on data from a single sensor or two used sensors according to the proposed fusion strategies. On the other hand, we investigate the training of uni-modal architectures as well as multi-modal architectures. We also evaluated the proposed fusion architectures on a real marine dataset that was collected by a vessel in the Finnish archipelago. The data represents images which are captured by RGB and IR cameras in different marine environmental conditions (i.e., weather conditions, light conditions, daytime/nighttime). To the best of our knowledge, no work has been done on studying the effectiveness of three different levels of fusion in marine environment. To summarize, the main contributions of this paper are in three-fold:

The remainder of the work is organized, as follows. Section 2 discusses some of the most important related works. The proposed architectures are presented in Sections 3–5. Sections 6 and 7 show the experimental setup and results of our implementations, respectively. Finally, we present our conclusions in Section 8.

#### **2. Related Work**

In this section, we briefly review the related work on infrared and visible image fusion and object detection using CNN. In addition, the vessel detection for maritime is also discussed.

**CNNs for fusion:** many image fusion techniques have been developed in recent years. The main idea of these techniques is obtaining salient features from input images and then combining them for generating a fused image [10]. Deep Learning (DL) is one of the widely-used approaches that has recently been used by theses techniques, since it can explore the features from the data efficiently [8]. It is able to obtain features from input images and then reconstruct a fused images with more details.

Multi-Scale CNN (MS-CNN) is one of these techniques that uses DL for performing pixel-level image fusion. It uses a proposal sub-network to perform target detection at multiple output layers, so that receptive fields match objects of different scales. These complementary scale-specific detectors are combined in order to create a strong multi-scale object detector. In [9], a middle fusion approach is proposed for fusing LiDAR and RGB data in order to classify objects in autonomous vehicle application. This approach first converts LiDAR point cloud data into depth map and then fed the data to a CNN for object classification. In a similar work [11], the dense depth map from LiDAR data and color imagery are fused for pedestrian detection while using CNN. Their results show that fusing LiDAR can improve the detection results. In another work, a DL-based fusion method [10] is presented to generate a fused image containing whole features from two sources IR and RGB images. We will describe the details of this method in Section 4.1.

DenseFuse [8] is another well-known DL-based fusion architecture for extracting and preserving most of the deep features of both RGB and IR images in a middle fusion fashion. In [1], a late fusion method is proposed based on the Probabilistic Data Association (PDA) [12] in order to produce object region proposals by fusing detection results from RGB, IR, radar and LiDAR. Then, a CNN is applied on the top of region proposals for classifying the interest objects within the regions. DyFusion [13] is a decision level fusion for maritime vessel classification. It first uses a CNN to generate the probabilities over maritime vessel classes for each input sensor. Subsequently, a fusion part updates the sensor probabilities by considering the contextual data.

PointFusion [14] leverages both image and three-dimensional (3D) point cloud data based on a late fusion architecture to perform target detection. The image data and point cloud data are independently processed by a CNN and then their results are combined to estimate object bounding boxes from image and point cloud data. The main contribution of PointFusion is using using heterogeneous network architectures. Moreover, the raw point cloud data is directly handled using a PointNet model, which avoids time consuming input pre-process such as quantization or projection.

**CNNs for object detection:** CNN were recently used in the development of object detection, as they are capable exploiting unknown structures in training data for discovering good representations [15]. The CNN-based object detectors are divided into two categories: two-stage detectors and one-stage detectors. Two-stage detectors employ an external module for generating interest object region proposals and their speed usually slower than one-stage detectors. In contrast, one-stage object detectors integrate region proposition and classification into one single stage. However, two-stage detectors usually have higher detection accuracy when compared to the one-stage detectors. Popular two-stage detectors include R-CNN [16], Fast/Faster R-CNN [17,18], and R-FCN [19]. Between one-stage detectors, SSD [20] and YOLO [21] are most common.

Region-based Convolutional Neural Network (R-CNN) [16], which leads to substantial gains in object detection accuracy. R-CNN first identifies region proposals and then classifies these regions into object categories or background using a CNN. One disadvantage of R-CNN is that it performs exhaustive search and proposes large number of regions from an image. Therefore, RCNN leads to time-consuming and energy-inefficient computation. The extension version of R-CNN is Fast R-CNN [17] which uses CNN to generate feature map straight from the input image instead of regions. Both R-CNN and Fast R-CNN use selective search for obtaining the region proposals. In order to reduce running time of Fast R-CNN, Faster R-CNN [18] omits the selective search method for generating object region proposals. Instead of using selective search, Faster R-CNN identifies the regions by using a separate network.

**Maritime vessel detection:** A few studies utilized object detection algorithms from waterborne images beyond maritime vessel detection from spaceborne imagery [22]. Some of these works have focused on classifying the interest objects from the background [23], others employed the Histogram of Oriented Gradients (HOG) approach using sliding-windows [24]. Recently, CNNs have been used for seaborne vessel detection. However, developing more new dataset and applications are necessary for autonomous maritime navigation. For instance, the Singapore Maritime Dataset is used in [25] for ship detection under a new proposed model, YOLO [21]. In [26], a contextual region-based convolutional neural network with multi-layer fusion is proposed for ship detection. It consists of a region proposal

network (RPN) and an object detection network with contextual features. Their results show that the additional contextual features provide more information for detection. However, this method can not detect small objects efficiently. In [27], an approach based on selective search is presented in order to extract the initial region proposals from RGB images. Subsequently, the initial proposals are filtered using the information from other sensors in order to find more dense proposals. Finally, a CNN is employed to identify the class of objects within the final proposals. The results are collected based on the marine data that were collected for the Advanced Autonomous Waterborne Applications Initiative (AAWA) project [28].

In [29], another novel dataset, SeaShips, consisting of a collection of in-shore and offshore ship images is introduced. Moreover, they used three object detectors (Faster R-CNN [18], SSD [20], and YOLO [21]) for detecting maritime vessels. In [30], a maritime vessel image dataset from a Vessel Tracking System (VST) is collected. This dataset contains authentic situations from traffic management operators. In addition, they proposed a SSD detector in order to identify vessels.

#### **3. The Proposed Early Fusion Architecture**

In this architecture, fusion happens at a very low abstraction level. As shown in Figure 1, the early fusion architecture concatenates RGB and IR images and produces a tensor with four channels (three channels from RGB and one channel from IR). This four-channel tensor is used as an input for a detector network. The intuition behind this is simple, since the features of the concatenated image should contain both information from RGB and IR. The detector produces Bounding Boxes (BBs) from the feature maps to localize the vessels. The localization is determined with a box that the top-left corner's coordinate (*x*1, *y*1) and bottom-right corner's coordinate (*x*2, *y*2). Moreover, each bounding box is associated with a confidence score *s*, which indicates how likely does the bounding box contain a vessel. The bounding boxes with the highest confidence are kept in order to filter by a Non-Maximum Suppression (NMS). NMS is a popular post-processing method in object detection methods [5,18] for filtering redundant bounding boxes and obtaining final detections.

**(4 channels input )**

**Figure 1.** An overview of the proposed early fusion architecture. (**A**) The 3-channel RGB input image and 1-channel IR image are concatenated. (**B**) Subsequently, the produced four-channel input data is processed by a detector in order to robustly detect vessels. (**C**) The output image consists of the predicted BBs and corresponding scores and labels.

#### **4. The Proposed Middle Fusion Architecture**

The middle fusion architecture consists of two layers, as illustrated in Figure 2. The first layer is a fuse layer that combines the information given by two RGB and IR cameras and constructs a fused image (Figure 2C). The fused image represents the thermal radiation information in infrared images and detailed texture information in visible images. Afterwards, a detector layer (Figure 2D) performs detection on the fused image in order to generate the object bounding box proposals.

**Figure 2.** An overview of the proposed middle fusion architecture. The original input images (**A**,**B**) are fused using by an image fusion method in order to provide complementary information for object detection. (**C**) The image fusion method can be one of the mentioned method in Sections 4.1–4.7. (**D**) Subsequently, the fused image is processed by a detector in order to detect and localize marine vessels. (**E**) The output image localizes the detected vessels with the corresponded scores and labels.

To generate the fused image in the fuse layer, we employed three DL-based image fusion methods (see Sections 4.1–4.3) and four traditional image fusion methods (see Sections 4.4–4.7). Here, we briefly review the tested image fusion methods, three DL and four traditional, which were evaluated in this work. The DL-based methods include: deep learning framework based on VGG19 and Multi-Layer (VGG-ML) [10], DenseFuse [8], and ResNet and Zero-phase Component Analysis-based method (ResNet-ZCA) [31]. The traditional fusion algorithms are categorized into two main groups: Multi-Scale Decomposition (MSD)-based methods [32] and Sparse Representation (SR)-based methods [33,34] according to the the fusion strategies. The MSD-based methods usually use different transform functions: pyramidal and discrete wavelet. The SR-based methods calculate the activity level of input images in a sparse domain. In this work, we utilized the weighted least square [32] as a MSD-based method and convolutional sparse representation [35] as a common SR-based method.

#### *4.1. Deep Learning Framework Based on VGG19 and Multi-Layers*

Deep learning framework based on VGG19 and Multi-layer (VGG-ML) [10] can combine the features from two source IR and RGB images and generate a fused image. For this purpose, the source images are firstly decomposed into base and details parts using the image decomposition method [36]. The base part of each source image contains the common features and redundant information and obtains it by the average filter. The details part represents the detail contents of source images and it produces by subtracting the base part from the source image. The base parts of both images are then fused by a weighted average strategy. For the detail parts, a pre-trained VGG19 network [37] obtains deep features from source images. Finally, the base and detail parts are added for creating a final output fused image.

#### *4.2. DenseFuse*

DenseFuse [8] is a deep network including three elements: encoder, fusion, and decoder. For testing the network, the encoder first extracts and preserves most deep features of both input RGB and IR images using DenseBlock [38] architecture. DenseBlock contains three cascaded convolutional layers. Subsequently, the fusion layer uses either additional fusion [38] or l1-norm fusion strategy for fusing the extracted features maps from both source images. Finally, the three convolutional-layered decoders receive the fused feature maps in order to create a fused image. For training the network, only encoder and decoder are employed to reconstruct the training images and fix weights of the network. In order to reconstruct the images, DenseFuse aims to reduce the *λ* weighted combination of pixel and structural similarity losses.

#### *4.3. ResNet and Zero-Phase Component Analysis-Based Fusion*

ResNet and Zero-Phase Component Analysis-based (ResNet-ZCA) method [31] has shown to be an efficient method for image fusion. Firstly, it employs ResNet [39] for extracting deep features from source images. Subsequently, ZCA [40] and l1-norm are used in order to project deep features into sparse domain. The initial weight maps are obtained by l1-norm. Finally, a bicubic interpolation is used to resize the initial weight maps to source image size. The final weight maps are generated by soft-max and the fusion image is reconstructed by final weight maps and source images.

#### *4.4. Visual Saliency Map and Weighted Least Square*

Visual Saliency Map and Weighted Least Square (VSM-WLS) [32] is a multi-scale fusion method that is based on WLS optimization and VSM. To perform Multi-Scale Decomposition (MSD), it first employs the rolling guidance filter [41] and Gaussian filter and decomposes both source IR and RGB images into base and detail parts. Afterwards, the fusion of base parts is carried by using a weighted average technique in order to enhance the fused image contrast. For fusing the detail parts, WLS optimization is used. Finally, inverse MSD is employed on the fused base and details parts to construct the output fused image.

#### *4.5. Cross Bilateral Filter*

Cross Bilateral Filtering (CBF) [42] is a non-iterative and local nonlinear method that combines an edge-stopping function with a low-pass filter for reducing the filter effect wherever the intensity between neighbouring pixels is large. It can filter the images while preserving the edges. Initially, CBF is applied to both RGB and IR source images to extract the base images. Subsequently, the detailed images are obtained by subtracting the base images from the original IR and RGB images. Finally, the fused image is obtained by multiplying the weights with input images, followed by a weight normalization.

#### *4.6. Convolutional Sparse Representation*

Convolutional Sparse Representation (ConvSR) [35] address the problem of SR-based image fusion methods by considering a global approach that aims the SR-based image fusion of the whole image rather than of local image patch windows. The global approach enhances the detail preservation and model sensitivity regarding mis-registration. ConvSR consists of hierarchical layers, where each layer includes an image decomposition to divide the input images into base and detail parts. The detail parts are combined using a choose-max strategy. An averaging strategy is applied in order to fuse the base parts and built the fused coefficient maps. The output fused image is built by combining the base and detailed layers.

#### *4.7. Guided Filtering Based Fusion Method*

Guided Filtering based Fusion (GFF) [36] method can generate a highly informative fused image based on a two-scale decomposition strategy. This strategy produces base and detail layers containing large scale variations in intensity and small scale details, respectively. Finally, a guided filtering-based weighted average technique is employed in order to make full use of spatial consistency for fusion of the base and detail layers.

**Figure 3.** An overview of the proposed late fusion architecture. (**A**) The input RGB image and (**B**) IR image are feed into the Detector1 and Detector2, respectively. (**C**) These detectors independently extract features from the corresponding input image. (**D**) The architecture concatenates outputs of detectors (*ORGB*,*OIR*), and then a final set of object proposals is obtained after none-maximum suppression. (**E**) The final output containing predicted BBs, which are associated with a category label and a confidence score.

#### **5. The Proposed Late Fusion Architecture**

Figure 3 demonstrates the proposed late fusion architecture. The late fusion architecture first combines the detection results from two detectors. These two detectors have similar architecture. One detector takes a RGB image as input and the other one takes the corresponding IR image as input. To be more specific, a separate detector is utilized in order to process each input camera image independently and extracts feature from the image. This process involves the estimation of the bounding box proposals, which indicate the objects' localization in the image. Subsequently, the output bounding boxes of two detectors (*ORGB*,*OIR*) are concatenated to explicitly capture complementary information of RGB and IR. In this case, fusion happens at the decision level. After that, the following steps are applied on the all boxes (*ORGB* + *OIR*) in order to generate final boxes and remove redundant detections, as follows:

$$f(b\_{\text{i}}, b\_{\text{best}}) = \begin{cases} 0, & \text{if } IoI < \mathfrak{a} \\ 1, & \text{if } IoI > \mathfrak{a} \end{cases} \tag{1}$$

where *α* is Intersection of Unit (IoU) threshold between two bounding boxes and it is determined experimentally. Based on a series of preliminary experiments, we got the best performance with *α* = 0.5. IoU is intersection of two boxes divided by their union.

$$IoI(b\_{i\prime}b\_{best}) = \frac{S\_{b\_i} \cap S\_{b\_{best}}}{S\_{b\_i} \cup S\_{b\_{best}}} \tag{2}$$

where *S<sup>b</sup>* represents the area of bounding box *b*.

**Figure 4.** An example of applying NMS in the proposed late fusion architecture: (**A**) the predicted BBs which their score is lower than 0.6 are removed and then (**B**) each box between the remaining boxes is assumed as an output box if IoU between ground truth BB and predicted BB is more or equal than 0.5.

#### **6. Experimental Setup**

#### *6.1. Datasets*

We collect a real marine dataset by a vessel in Finnish archipelago for evaluating our proposed fusion architectures. The dataset is recorded by two sensors continuously, providing data from various environmental and geographical scenarios. This sensor system includes RGB (visible spectrum) and IR (thermal) camera arrays, providing output that can be synchronized and stitched to form panoramic images. The individual visible cameras have full HD resolution while the thermal cameras have VGA resolution. Both camera types have a horizontal field of view of approximately 35 degrees. For image alignment in this sensor set, the registration parameters are manually determined by finding corresponding features in calibration images and minimizing alignment mismatch. Therefore, our dataset contains well-aligned IR/RGB images. The images were sampled one frame per second in and stored in MPEG format. The images show maritime scenarios under different illumination conditions with various marine vessels. We manually annotated all vessels (passenger vessel, motorboat, sailboat, or docked vessel) within each RGB sequence with a bounding box as accurately as possible. However, all of the vessels have a general label "Vessel" in our datatset. The bounding box should contain all pixels that belong to that object and, at the same time, be as tight as possible. In addition, two different scenarios are proposed in order to evaluate the proposed architectures in different light condition, time imaging and location.

**Scenario1**: the training dataset is collected by two visible and infrared cameras at daytime. In this scenario, the training dataset consists of 7250 pairs of well-aligned multispectral images captured by cameras. For evaluation, a separate test dataset is gathered in the same light and weather condition contains 1750 RGB/IR pair images. Figure 5a demonstrates a sample of RGB images and corresponding IR in this scenario. The number of vessels in the training and test datasets is determined in Table 1.

**Scenario2**: RGB and IR images are collected by a vessel operating near the harbour at nigh time. This data represent a challenging data (dark and oversaturated areas) in marine environment. The source videos for generating training and test images are different. The training and test datasets consist of 2250 and 1000 pair RGB/IR images, respectively. Table 1 shows the number of vessels in each dataset. Furthermore, Figure 5b illustrates an IR/RGB pair of a sample of our data in this scenario.

The original size of all images is 3240 × 944 pixels for both scenarios. To reduce the computation time, we re-sized the original images into 1200 × 400 pixels.

**Table 1.** Number of vessels in our training and testing marine datasets for each Scenario.

**Figure 5.** Example of RGB and InfRared (IR) pair images in the real maritime dataset at (**a**) Scenario1 and (**b**) Scenario2.

#### *6.2. Implementation Details*

Here, we give more information regarding the method parameters. The parameter setting of the proposed (1) image fusion methods in the middle architecture and (2) CNN-based detector in all architectures are as follows:

**Image fusion methods**: we selected all parameters of the image fusion methods based on the existing works which are described in Section 4. VGG-ML fuses the detailes parts by using VGG-19 [37] with four relu layers. The weight values for pixel in two base part images *α*<sup>1</sup> = 0.5 and *α*<sup>2</sup> = 0.5 in VGG-ML. DenseFuse is pre-trained on MS-COCO [43] and utilizes two methodologies for fusion: addition and l1-norm. DenseFuse achieves the minimum pixel and structural similarity losses when *λ* is 100. For ResNet-ZCA, we used ResNet50 with l1-norm. ResNet50 is pre-trained by ImageNet [44]. In VSM-WLS, the initial spatial weight, *σ<sup>s</sup>* , is 2. The number of decomposition levels *N* is 4 and *λ* = 0.01. CBF uses the neighborhood kernel with 11 × 11 size, as it can achieve good enough fusion results [42]. The value of *σ<sup>s</sup>* and *σ<sup>r</sup>* are 1.8 and 25, respectively. Moreover, the parameter *λ* is fixed at 0.01 in ConvSR. In the GFF experiment, the parameters of the guided filter are set as *r*<sup>1</sup> = 45, *e*<sup>1</sup> = 0.3, *r*<sup>2</sup> = 7 and *e*<sup>2</sup> = 10−<sup>6</sup> . All of the image fusion methods require the grayscale images transformed from the input RGB images except DenseFuse and VSM-WLS, .

**CNN-based detectors**: we use Faster R-CNN as a detector in all proposed architectures. The CNN parameter are chosen based on several experimental results. Faster R-CNN is trained for 900 k iterations with a learning rate of 0.0003 and then 1200k iterarions with a learning rate of 0.000003. We use 4 sub-octave scales (0.25, 0.5, 1.0, 2.0) and three aspect ratios [0.5, 1.0, 2.0] mainly motivated by handling small objects on this dataset.

Since Microsoft COCO dataset [43] consists of 3146 images of marine vessels, the Faster R-CNN is pre-trained on it to learn more good feature representation. Subsequently, the model is fine-tuned on our data. We utilize different source videos to train and test architectures. These fixed parameter setting can obtain good results for our experiments done in this work.

#### **7. Experimental Results**

In this work, three multi-modal architectures were considered for vessel detection: early fusion, middle fusion, and late fusion. In addition, two uni-modal architectures are proposed, which utilized RGB or IR camera images. We have done three experiments: (1) evaluation of seven image fusion methods in the middle fusion architecture, (2) evaluation of all fusion architectures, and (3) a visual comparison between all architectures in each scenario.

#### *7.1. Comparison of Image Fusion Methods*

In the propose middle fusion architecture, an image fusion method is first employed to combine source RGB and IR images and produce a fused image (see Sections 4.1–4.7). Subsequently, a CNN is applied on the obtained fused image for detection. Therefore, the image fusion method provides an essential functionality in our proposed middle fusion architecture. For this reason, we first evaluated the performance of three DL-based image fusion methods and four traditional methods. The details of our experiment are introduced in Section 6.2. These methods are compared with six common assessment metrics to conduct qualitative and quantitative experiments. These metrics include:

Figures 6 and 7 demonstrate the average values of performance metrics for whole test dataset in two scenarios. In Scenario1 (Figure 6), the results show that DL-based fusion methods perform better than traditional methods with the larger values of *FMIw*, *FMIdct*, and *SSIM*. The reason is these methods (VGG-ML, DenseFuse, and ResNet-ZCA) can extract more structural and rich features that are based on their DL architectures. Between these DL-based methods, ResNet-ZCA has the highest value of *FMIw*, *FMIdct*, and *SSIM*. However, DenseFuse provide more natural results and contain less artificial noise as it has the minimum values of *NAB*/*<sup>F</sup>* , *QAB*/*<sup>F</sup>* , *EN* and *SCD*. Between traditional methods, GFF can achieve more complementary information in the fused image, since it has the maximum value of *FMIw*, *FMIdct*, and *SSIM*.

Figure 7 shows the average values of six quality metrics for Scenario2. We can observe that DL-based method is roughly more natural and less noise than other traditional methods. Furthermore, the results represent DenseFuse can generate the fused image with less artificial information and noise as the value of *NAB*/*<sup>F</sup>* is low. However, ResNet-ZCA provide more structural information and features, as it has the highest value of *FMIw*, *FMIdct*, and *SSIM*. GFF performs betters than other traditional image fusion methods in terms of *FMIw*, *FMIdct*, and *SSIM*. This is because GFF can effectively keep the contrast in the fused image.

**Figure 6.** The average values of six quality metrics for test images obtained by the deep and traditional methods in Scenario1.

**Figure 7.** The average values of six quality metrics for test fused images obtained by the deep and traditional methods in Scenario2.

Moreover, we performed a visual comparison between all image fusion methods for an example test image in each scenario. In the scenario1, the obtained fused image by DL-based method contains more frequency details and edge preservation (Figure 8A–D). The fused image that is generated by VSM-WLS, CBF, ConvSR, and GFF includes more artificial noise and their saliency features are not clear. CBF and ConvSR produce the fused images with more artifacts as well. On the contrary, the fused images obtained by VGG-Ml, DenseFuse, ResNet-ZCA and VSM-WLS look more natural and less noise. Generally, the obtained results of these DL-based methods are roughly more clear than other traditional methods in Scenario1.

Figure 9 shows the fused image obtained by DL and traditional image fusion methods in the Scenario2. From the Figure 9A–E, it is observed that VGG-Ml, DenseFuse, ResNet-ZCA, and VSM-WLS provide a more pleasing image with clear texture details. From the red box (part of a land), it is observed the fused image by VGG-Ml contains less noise, and details are more clearer than other image fusion methods. In contrast, CBF, ConvSR, and GFF (Figure 9F–H) produce results with more noise, color distortion and contrast loss.

*Remote Sens.* **2020**, *12*, 2509

**Figure 8.** Qualitative results of the fused image in Scenario1 by (**A**) VGG-ML, (**B**) DenseFuse-add, (**C**) DenseFuse-l1, (**D**) ResNet-ZCA, (**E**) VSM-WLS, (**F**) CBF, (**G**) ConvSR, and (**H**) GFF on the original RGB and IR images.

**Figure 9.** Qualitative results of the fused image in Scenario2 by (**A**) VGG-ML, (**B**) DenseFuse-add, (**C**) DenseFuse-l1, (**D**) ResNet-ZCA, (**E**) VSM-WLS, (**F**) CBF, (**G**) ConvSR, and (**H**) GFF on the original RGB and IR images.

**Processing Time:** Table 2 shows the running time (second) of all image fusion methods for one image. The tested system specification is: Intel(R) Core(TM) i7-4702MQ CPU @ 2.20 GHz×8 CPU with 15.4 GB RAM. The running time for obtaining the fused image by ResNet-ZCA is 4.9 s per image. ResNet-ZCA has the minimum time between DL-based methods. In addition, GFF can generate a fused image in 0.4 s that is lower than ResNet-ZCA.

**Table 2.** The running time (seconds) of the deep and traditional image fusion methods for one image.

#### *7.2. Multi-Modal Architectures vs. Vni-Modal Architectures*

We compared the fusion architectures for the test dataset in terms of Average Precision (AP) as a main detection accuracy metrics. For this purpose, we measured the IoU of detected bounding boxes and matching those from ground truth annotations. A detected bounding box result is counted as a true positive if the IoU with a ground truth one exceeds 50%. Unmatched detected bounding boxes are counted as false positives and unmatched ground truth ones are counted as false negatives.

Table 3 shows that AP for the proposed architectures in each scenario. The best results are highlighted in bold. This results show the effect of the fusion on the object detection performance, as we compared uni-modal and multi-modal architectures. It is observed from the result, the multi-modal middle architecture generates reliable detection results (bold font in Table 3) for both scenarios (scenario1:79.1% and scenario2:61.6%), as it can provide complementary information when compared with the uni-modal architectures. However, the performance can be improved when the dataset contains more bigger targets. Our dataset consists of large amount of small targets which occupying areas lower than 16 by 16 pixels. Detecting very small objects with a few pixels is still challenging because of less information being associated with them.

In addition, the results show that uni-modal RGB-based architecture can provide higher accuracy in comparison with uni-modal IR-based architecture. For instance, the accuracy of uni-modal RGB-based architecture is 9.0% and 9.7% more than the uni-modal IR-based architecture for scenario1 and 2, respectively. This is because it can learn richer features from color images than infrared images. Moreover, the results show that DenseFuse totally have higher accuracy in comparison with other middle-fusion architectures.

**Table 3.** Average Precision (AP) (in %) on the test dataset of two scenarios.

### *7.3. Qualitative Results*

Figure 10 demonstrates an examples of the detection results from the visible-only architecture, infrared-only architecture and multi-modal architectures in each scenario (day-time and night-time). We observe that the proposed fusion architectures is better at the detection of objects than the uni-modal architectures. Note that, because the fusion architectures can integrate information from both color and infrared images. The fusion architectures successfully detected the size/location of the bounding boxes. In the third row, our middle- fusion architecture has detected marine vessels that other architectures have missed. Moreover, the middle-fusion architecture is able to detect small objects

with a few pixels as shown in Figure 10 and many of them are detected by our framework. It shows the generalisation capability of the proposed middle-fusion architecture and indicates its potentials in executing two-dimensional (2D) object detection in real situations beyond a pre-designed dataset.

**Figure 10.** Qualitative vessel detection results for (**A**) Scenario1 and (**B**) Scenario2 based on uni-modal based on RGB, uni-modal based on IR, multi-modal early fusion, multi-modal middle fusion, and multi-modal late fusion architectures. The ground truth bounding boxes are shown as green rectangles. Predicted boxes by the architectures are depicted as red bounding boxes. Each output box is associated with a category label and a score value in [0, 1].

#### **8. Conclusions**

In this paper, we proposed three image fusion architectures for vessel detection in marine environments. The architectures can combine the thermal radiation information on infrared images and the texture detail information on visible images. They also utilized a simple fast CNN, Faster R-CNN, in order to carry out the final detection task. The evaluation on our real marine dataset show that the proposed middle-fusion architecture is able to detect the vessel at the state of the art accuracy.

For future work, we plan to improve the detection network of these architectures in order to address the detection problem of very small objects (less than 16 by 16 pixels) in our data. We will investigate the effect of using transfer learning and domain-specific data on the detection performance. We also plan to extend our fusion framework by considering other common sensors in autonomous vessels, such as LiDAR and radar, besides IR and RGB cameras.

**Author Contributions:** F.F. conceived and designed the methodology; performed the experiments; analyzed the data; wrote the paper. J.H. supervised the study and reviewed this paper. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work is funded by the Tekes (Finnish Funding Agency for Technology and Innovation) as a part of autonomous Ships and Machines project at Turku university.

**Acknowledgments:** Computational resources were provided by CSC-IT Center for Science Ltd., Espoo, Finland. **Conflicts of Interest:** The authors declare no conflict of interest.

### **References**

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## *Article* **ABOships-An Inshore and Offshore Maritime Vessel Detection Dataset with Precise Annotations**

**Bogdan Iancu \*,†, Valentin Soloviev † , Luca Zelioli † and Johan Lilius †**

> Faculty of Science and Engineering, Åbo Akademi University, 20500 Åbo, Finland<EMAIL_ADDRESS>(V.S<EMAIL_ADDRESS>(L.Z<EMAIL_ADDRESS>(J.L.)

**\*** Correspondence<EMAIL_ADDRESS>
† Current address: Åbo Akademi, Agora, Informationsteknologi, Vattenborgsvägen 3, 20500 Åbo, Finland.

**Abstract:** Availability of domain-specific datasets is an essential problem in object detection. Datasets of inshore and offshore maritime vessels are no exception, with a limited number of studies addressing maritime vessel detection on such datasets. For that reason, we collected a dataset consisting of images of maritime vessels taking into account different factors: background variation, atmospheric conditions, illumination, visible proportion, occlusion and scale variation. Vessel instances (including nine types of vessels), seamarks and miscellaneous floaters were precisely annotated: we employed a first round of labelling and we subsequently used the CSRT tracker to trace inconsistencies and relabel inadequate label instances. Moreover, we evaluated the out-of-the-box performance of four prevalent object detection algorithms (Faster R-CNN, R-FCN, SSD and EfficientDet). The algorithms were previously trained on the Microsoft COCO dataset. We compared their accuracy based on feature extractor and object size. Our experiments showed that Faster R-CNN with Inception-Resnet v2 outperforms the other algorithms, except in the large object category where EfficientDet surpasses the latter.

**Keywords:** maritime vessel dataset; ship detection; object detection; convolutional neural network; deep learning; autonomous marine navigation

### **1. Introduction**

Maritime vessel detection from waterborne images is a crucial aspect in various fields involving maritime traffic supervision and management, marine surveillance and navigation safety. Prevailing ship detection techniques exploit either remote sensing images or radar images, which can hinder the performance of real-time applications [1]. Satellites can provide near real-time information, but satellite image acquisition, however, can be unpredictable, since it is challenging to determine which satellite sensors can provide the relevant imagery in a narrow collection window [2]. Hence, seaborne visual imagery can tremendously help in essential tasks both in civilian and military applications, since it can be collected in real-time from surveillance videos, for instance.

Ship detection in a traditional setting depends extensively on human monitoring, which is highly expensive and unproductive. Moreover, the complexity of the maritime environment makes it difficult for humans to focus on video footage for prolonged periods of time [3]. Machine vision, however, can take the strain from human resources and provide solutions for ship detection. Traditional methods based on feature extraction and image classification, involving background subtraction and foreground detection, as well as directional gradient histograms, are highly affected by datasets exhibiting challenging environmental factors (glare, fog, clouds, high waves, rain etc.), background noise or lighting conditions.

Convolutional neural networks (CNNs) contributed massively to the image classification and object detection tasks in the past years [4–8]. They incorporate feature extractors

**Citation:** Iancu, B.; Soloviev, V.; Zelioli, L.; Lilius, J. ABOships-An Inshore and Offshore Maritime Vessel Detection Dataset with Precise Annotations. *Remote Sens.* **2021**, *13*, 988. https://doi.org/10.3390/ rs13050988

Academic Editors: Pedro Melo-Pinto

Received: 4 February 2021 Accepted: 1 March 2021 Published: 5 March 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

and classifiers in multilayer architectures, whose number of layers regulate their selectiveness and feature invariance. CNNs exploit convolutional and pooling layers extracting local features, and gradually advancing object representation from simple features to complex structures, across multiple layers. CNN-based detectors can subtract compelling distinguishable features automatically unlike more traditional methods which use predefined features, manually selected. However, integrating ship features into detection proves to be challenging even in this context, given the complexity of environmental factors, object occlusion, ship size variation, occupied pixel area etc. This often leads to unsatisfactory performance of detectors on ship datasets.

To address ship detection in a range of operating scenarios, including various atmospheric conditions, background variations and illumination, we introduce a new dataset consisting of 9880 images, and annotations comprising 41, 967 carefully annotated objects.

The paper is organized as follows. Section 2 describes related work, including notable results in vessel detection and maritime datasets comprising waterborne images. Section 3 describes data acquisition, dataset diversity, dataset design and our relabelling algorithm along with basic dataset statistics based on the final annotation data. In Section 4, we discuss evaluation criteria and present experimental results; we investigate four CNNbased detectors and discuss the feature extractors and object size effect on the performance of the detectors. Section 5 provides a qualitative overview of the experimental results. In Section 6, we provide a brief analysis of our dataset specifications in comparison with other similar datasets. Conclusions are presented in Section 7.

#### **2. Related Work**

#### *2.1. Object Detection*

Object detection is one of the fundamental visual recognition problems where the requirement is to predict whether there are any objects from given categories in an image and provide their location (bounding boxes or pixel-level localization in case of instance segmentation), if any are found. Generally, this is achieved by extracting features in an image and matching them against features from trained images. Traditional approaches use sliding windows to generate proposals, then visual descriptors to generate an embedding, which are subsequently classified (such as SVM, bagging, cascade learning and AdaBoost). Traditional algorithms with best performance focus on carefully designing the descriptors for extracting the features (SIFT, Haar, SURF). However, since 2008, more and more limitations of this approach became evident [7]. We list below the most notable ones:

In the early 2010s, deep learning approaches came to prominence and started replacing the traditional ones. Object detection networks can be roughly categorized into 2 types: onestage detectors and two-stage detectors. The structure of the latter resembles traditional object detectors in that they generate proposal-regions and then classify the proposals, while the former considers positions within an image as potential objects and attempts to classify them immediately. The traditional approach of sliding windows for proposal generation is still used in CNNs, but other notable advances emerged, which allow for more efficient proposal generation, such as anchor-based and key-point approaches (CenterNet being one of the more notable examples of the kind) [7].

However, the key difference between traditional object detection and CNNs stems from the manner in which visual descriptors are generated. In deep learning, instead of creating visual descriptors by hand, convolutional layers perform this role. Instead of defining feature extractors by hand, basic CNNs train multiple convolutional layers to extract both high- and low-level features, which are then classified with the help of fully-connected layers. The resulting network essentially solves all the main limitations of a traditional approach, but the trade-off is that it requires a significantly larger number of training images for hyperparameter optimization [7,8].

While the requirement of a large number of training samples can prove to be a large obstacle, one of the benefits of CNN-based models is that they can be generalized into other fields with similar characteristics with the help of transfer learning. By training a model on a specific dataset, the backbone of the model can be later used to extract features in other tasks with similar features. For this reason, the aim of recent CNN-models was to be as generic as possible, since with the help of transfer learning, they can be specialized for the field of interest. The challenge, however, appears when those generic models are not suitable feature extractors for a new field and there is not enough data to train them [6]. For those specific cases, the only viable solution is creation of new datasets.

#### *2.2. General Object Detection Datasets*

The two main reasons for the remarkable progress computer vision made in the past decades are the availability of large-scale datasets and powerful GPUs that made it possible for deep learning to take off considerably [9]. Deep learning made notable contributions to the field of computer vision, the tasks of image classification and object detection being in the forefront of research areas that benefited from it. International competitions such as ILSVRC, PASCAL VOC, and Microsoft COCO motivated the community tremendously, each of their contributions offering large-scale datasets that have been exploited ever since. These general object detection datasets have been extensively used for object detection with deep neural networks. They are essential for testing and training computer vision algorithms. We will discuss below some of the most prominent general-purpose object detection datasets.

Microsoft COCO [10] provides a selection of 330,000 images with a number of 2.5 million of labelled object instances, over 91 object classes. The dataset labeling used perinstance segmentation to ensure precise object localization. Two crucial aspects of the dataset are that it exhibits abundant contextual information and images contain multiple objects per image.

The ImageNet Large Scale Visual Recognition Challenge (ILSVRC) ran annually for a number of years and was established as one of the typical benchmarks for object classification and detection. The Imagenet dataset [4], the foundation of the challenge, is an image collection based on the WordNet hierarchy [11], which provides on average 1000 manually verified images for every synset (synonym set) in the hierarchy.  The dataset consists of over 14 million images, of which over 14 million were annotated to denote what objects are present in the image and, for over a million of them, bounding boxes are provided too.

Pattern Analysis, Statistical Modelling and Computational Learning (PASCAL) Visual Object Classes (VOC) is a prominent project in the computer vision community, which provided publicly available image datasets including ground truth annotations and standardized evaluation metrics. These datasets were exploited as part of a number of challenges on various tasks such as: classification, detection, segmentation, etc. The greater number of scientific publications regarding object detection use the PASCAL VOC challenges to benchmark their proposed algorithms. The reason is that these challenges introduced a number of evaluation methods: bootstrapping, to decide significant differences among algorithms, a normalised average precision across classes, etc. The dataset released by last PASCAL VOC challenge includes 11,530 images with 27,450 annotated objects over 20 classes. Table 1 shows a variety of object detection datasets, with their total number of images and clasess. We can notice that ImageNet is by far the largest of the ones mentioned in the table, encompassing the greater number of total images and classes.

**Table 1.** Different object detection datasets comprising various object classes, with their corresponding annotations.

Of the general-purpose object detection datasets, in Table 1, the total number of maritime vessels included is limited, only Microsoft COCO comprising a considerable amount of vessels, 3146. All vessel counts can be found in Table 2.

**Table 2.** Maritime vessel instances in general object detection datasets.

#### *2.3. Maritime Vessel Detection Datasets*

Maritime vessel detection from satellite imagery was employed in many studies, over the past 40 years, a review from 2018, [12], gathering a number of 119 papers regarding ship detection and classification only from optical satellites.  Some studies proposed algorithms utilizing the idea of background subtraction and detection of the foreground in maritime images. This class of techniques is predominantly used in surveillance applications due to their ability to perform well with unexpected changes in illumination, frequency or background noise [13]. Other studies proposed solutions for ship detection based on the Histogram of Oriented Gradients (HOG) and sliding windows [14].

However, since the bloom of deep learning in the past 15 years, CNNs were employed in ship detection from waterborne images. Even so, datasets of seaborne images are scarce, the most notable ones we briefly discuss below.

The Singapore Maritime Dataset, introduced in [15] consists of 80 videos recorded during daytime and nighttime, and provides ground truth labels for every frame of every video, comprising bounding-boxes and object classes for the corresponding boundingboxes. The annotations for the Singapore Maritime Dataset include 10 object classes, of which 6 ship types. This dataset is used for ship detection employing the YOLO v.2 algorithm [16].

Another recent ship dataset, SeaShips [3], consists of over 31,455 inshore and offshore images of ships, comprising 6 ship types. In [3], they employ three object detectors (Faster R-CNN [17], SSD [18] and YOLO [16]) to detect ships.

One of the most recent datasets published is MCShips [19], comprising a number of 14,709 images of ships, whose annotations cover 6 warship classes and 7 civilian ship classes. In [19], they also use the object detection algorithms above (Faster R-CNN [17], SSD [18] and YOLO [16]) to evaluate the dataset over the 13 ship classes.

We compared our ABOships dataset against other existing ship datasets. Table 3 illustrates the main differences. Our dataset has the smallest number of images (9880) amongst the four datasets, however it contains a great number of annotations (41, 967)

given the image total, which shows it represents well real scenarios of maritime imagery, taking into account the fact that it includes on average more than 4 annotated objects per image.

**Table 3.** Comparison of ABOships with other maritime datasets.

#### **3. Materials and Methods**

#### *3.1. Camera System*

The dataset was acquired from a set of 135 videos, collected from a sightseeing watercraft, by a camera with a field of view of 65◦ and stored in FullHD (1920 × 720) resolution at 15 FPS in MPEG format. The route of the watercraft extended from the city of Turku to Ruissalo in South-West Finland, the videos comprising the urban area along the Aura river, the port and the Finnish Archipelago, for a duration of 13 days (26 June 2018–8 July 2018). The watercraft ran each day in a timeframe between 10.15 and 16.45. The videos were captured into 30-min long periods consisting of footage from the route that the watercraft took. While the route remained largely the same, the data contains a variety of typical maritime scenarios in a range of weather conditions.

In addition to camera video data, the platform had a LiDAR attached to it (SICK LD-MRS, FoV 110 degrees, 2 × 4 planes, up to 300 m detection, at 5 Hz). The data from the LiDAR was captured alongside the video at a rate of 5 entries of up to 800 points per 0.2 s. Given the utilized LiDAR had a detection range of up to 300 m, it was very useful for detecting other objects in the harbor environment. Due to having only 2 times 4 lasers in the height direction however, the provided data was not reliable enough for discerning the nature of the object (i.e., what object was detected). It was useful however to determine distances to the objects perceived in the videos. For the purpose of creating the dataset presented in this paper, we used the LiDAR data to filter out video segments that were captured in the harbor area (usually the ones that had too many points for a prolonged period of time).

To evaluate the models, we acquired 9880 image photos from the videos. First, we annotated all images with 11 categories: seamarks, 9 types of maritime vessels, and miscellaneous floaters. In a second round, we relabelled all the inconsistencies we found, using an algorithm based on the CSRT tracker [20].

#### *3.2. Dataset Diversity*

Maritime environments are inherently intricate, hence a range of factors have to be accounted for when desinging a dataset. Dataset design must ensure that the dataset characterizes well vessels in the environment. Of course, data augmentation methods can be considered for reproducing certain environmental conditions, however authentic conditions may be difficult to anticipate.

*Background variation.* Particular object detection tasks are more prone to be affected by changes in the background of the picture. For instance, facial recognition is less susceptible to background variations, because given the similar shape of most faces, it is easier to fit them into bounding boxes in a congruous manner. However, the shapes of maritime vessels are highly heterogeneous, making them more difficult to separate from the background due to a potentially vast background information in the bounding box. The accuracy of ship detection would be significantly affected if background information were classified as ship features. Figure 1 illustrates the background variation of images in our datasets, including urban landscapes and an open sea environment.

**Figure 1.** Example image of background variations in the ABOships dataset: (**a**) View of maritime vessels on Aura river including the urban landscape; (**b**) View of a maritime vessel in the Finnish Archipelago.

*Atmospheric conditions.* Atmospheric conditions were specific to Finnish summers, with very sunny periods, alternating with rainy intervals and cloudy skies. The dataset includes a variety of images of different atmospheric conditions throughout a day.

*Illumination.* Lighting variations can significantly impact image capture. Illumination throughout the day, in various geographical areas and with specific daylight hours in a given region can dramatically influence image detection.

*Visible proportion.* A great number of the images in our dataset consists of moving ships, with objects being only partially captured in the camera field of view. However, they still represent objects that were annotated since one has to detect them as well. The annotation should comprise different visible proportions of the maritime vessels.

*Occlusion.* Due to the fact that our dataset has been captured in an open sea environment, in the harbor area and also comprises urban landscapes, there are many occasions when maritime vessels occlude each other or occlude other objects in the environment in the harbor area or in the urban landscape. In a subset of pictures especially in the harbor area, there is significant occlusion due to a high number of maritime vessels in the proximity of each other. Two examples of occlusion are shown in Figure 2.

**Figure 2.** Example image of a occlusion: (**a**) Boat in front of a militaryship; (**b**) Several sailboats occluding each other while docked, on the right half of the image.

*Scale variation.* Detection of small object can prove to be quite difficult, especially in a complex environment like the sea, ships that occupy a small pixel area in the picture can be confused with other objects in the background. Maintaining a high level of detection for ships requires including several scales for ships sizes in the dataset. For more information regarding the annotation and the size of the bounding boxes, please refer to Section 3.4.

Figure 3 illustrates a sailboat from two different perspectives: a lateral and a frontal view, which shows a variation in both occupied pixel area, but also the visible proportion.

**Figure 3.** Example image of a sailboat, view from two perspectives: (**a**) Lateral; (**b**) Frontal.

#### *3.3. Dataset Design*

The raw data acquired from the camera on the sightseeing watercraft is captured as MPEG videos, with 720 p resolution at 15 FPS . The videos include some footage exhibiting content that is irrelevant for the scope of vessel detection (especially footage captured when the watercraft was docked, either at the start of its route on the Aura river or at the Port of Turku) or sensitive content, such as faces of people. To address the latter issue, we performed face detection on all videos and blurred all detected faces. Addressing the former issue on the other hand, required additional data from the LiDAR.

In a maritime environment, LiDAR data is relatively sparse, authors of this study observed that a high number of points detected for a prolonged duration correlates with the watercraft being docked in the harbor. By setting a point threshold to detect these (docked/harbor) cases, we were able to filter them out in their majority and extract only the images regarding mostly the maritime environment. The images were extracted at an interval of 15 s (one image every 225 frames) and still contained some images captured during docking, but most of them were facing outwards from the harbor, so the images captured in this manner still contain useful maritime data. As a result we acquired 9880 images in the maritime environment.

The acquired images were subsequently separated into workpackages in such a manner that chronologically adjacent pictures were separated into different workpackages. The workpackages were then manually labelled by different annotators. After the initial labelling was completed, we used the CSRT tracker [20] to combine labels of the same object into traces, i.e., a collection of chronologically adjacent images containing a bounding-box for that object. Due to inaccuracies in the tracking process and discrepancies in labelling, the produced traces were not always accurate. After viewing the labels in these traces, we identified the main causes for discrepancies in labelling, which were mainly caused by different interpretations of label annotations. We refined those annotations to eliminate the discrepancies and separated the data into a second collection of workpackages that were provided to annotators, who then relabelled the data, according to refined annotations. After the relabelling was completed, the images and their refined labels were compiled into a dataset of maritime images with refined annotations.

#### *3.4. Annotation*

To perform the annotation task, we first investigated the captured videos and identified the vessel types that appeared most often. Due to the fact that the videos were captured at locations with a significant number of passenger ships, there is a certain level of bias for labelers towards those types of ships. This is different from the Seaships database, for instance, which comprises a higher variety of cargo ships. For the purposes of future use in machine vision, rather than using maritime terminology as such (depicting ship scale and purpose), we selected labels that had some clearly distinct visual characteristics. A visual representation of the labels is illustrated in Figure 4. The label categories are discussed below, with more specific details for every category:

• boat—rowing boats or oval-shaped boats (from a lateral perspective), or small-sized boats, visual distinction – rowing-like boats even if they possess engine power;

**Figure 4.** Example images of annotated objects in the ABOships dataset: (**a**) boat, (**b**) cargoship, (**c**) cruiseship, (**d**) ferry, (**e**) militaryship, (**f**) miscboat, (**g**) miscellaneous (floater), (**h**) motorboat, (**i**) passengership, (**j**) sailboat and (**k**) seamark.

#### *3.5. Relabelling Algorithm*

The labelling was performed by multiple annotators with different backgrounds, hence some label types were interpreted differently among them. To increase the consistency of labelling, we used the continuous nature of the raw data by tracking the labels between frames using the CSRT tracker [20]. For every labelled frame, a tracker instance was created. The aim was to track an object until the next labelled frame. At that point, the existing traces would be mapped onto the labels of the new frame, based on the *IoU* metric. During this mapping, it was assumed that labellers would not confuse seamarks with

vessels, hence ship labels were not mapped onto seamarks or vice versa. More importantly, previous labels were not taken into consideration, so even if annotators gave the same object conflicting labels in different frames, these labels would still belong to the same trace as long as the tracker could identify them. For cases where the mapping could not be found, the trace would assign a new label, <*Unlabeled*>, to denote that even though nothing was labeled in that specific case, the tracker indicated that the object should belong to the trace.

After a certain number of frames, either the tracker would lose the object (the most common reasons for this being object occlusion, or due to the object being either too far or exiting the frame altogether) or the tracker would have none of the defined labels mapped to it enough times (which would mean it most likely drifted onto another object). In both of those cases, the tracker was stopped and the resulting trace was saved to a file for further processing as described below.

To reduce the number of errors caused by occlusion and the tracker drifting towards other objects than the current object of interest, we performed a second tracking in the backwards direction. By comparing labels identified in the traces acquired from tracking videos in both directions, one could detect situations where traces could not be mapped onto each other. Those cases signify that the tracker was either occluded or drifted to another object, so traces required to be split into smaller sequences still, until no more conflicts could be detected.

The resulting traces (after the backwards tracking) were provided as batches for relabelling. Traces containing a single entry were batched together with other singular traces from the same category. This setup was done with the purpose of preventing and removing accidental labels (mislabeling), while, at the same time, providing more information about the objects being annotated. This allowed us to accurately label even the objects at a longer distance as a consequence of tracking history. Traces obtained in this manner were then provided for relabelling as a collection of labels belonging to the same trace and annotators were asked to refine the labels so that labelling would be consistent with the labelling specifications. Singular entries that did not belong to any trace were subsequently batched together with other objects of the same category. The process described above is illustrated in Figure 5, while the relabelling software application is depicted in Figure 6.

**Figure 5.** The video collection was separated into 48 workpackages of images (1), which were labelled in an initial labelling step (2). Using the OpenCV Tracker, the objects were tracked across frames to produce traces (3) and then relabelled to fix inconsistencies and fill in the labels that might have been skipped (4). The resulting labels were then compiled into the maritime imagery dataset (5).

**Figure 6.** The relabelling process utilized our relabelling software application. Its GUI (graphical userinterface) shows the annotator traces of tracked images between annotation frames (1). The annotator is required to either relabel every instance by selecting the correct label from the right panel, or edit an annotation (by selecting a label that emerged distinct from others (2)) and change the label of each image individually and possibly fix the bounding box to fit the object more tightly (3). Special attention was required in certain situations when the tracker would drift onto other objects, in which case that particular entry of the trace might have had a different label from the rest (4). When all labels belonging to a trace were verified and steps (1)–(4) were completed (5), the changes were saved into a new file and the annotator was provided with the next trace.

#### *3.6. Dataset Statistics*

Table 4 shows the number of images of each category in our dataset and the number of annotations. The column Images represents the number of images that contain that particular object class and then the percentage of images that comprise that class follows. Then the column Objects represents the number of annotations for that particular class in the dataset, along with the percentage of objects annotated for that specific class out of all the annotated objects in the dataset. One can notice from Table 4 that the highest representation of labels in the images from ABOships dataset is reached by three categories: motorboats (present in 41.11% of the images), sailboats (present in 38.88% of the images), and seamarks (present in 37.89% of the images). Conversely, the lowest representation is registered for cargoships (in 1.58% of the images) and miscellaneous floaters (in 1.30% of the images).

Moreover, Figure 7 illustrates the distribution of annotated objects in our dataset based on occupied pixel area at log2-scale, for every object category, and separates every object category by size in small, medium and large objects based on the Microsoft COCO variants (small: log2(area) < 10, medium: 10 < log2(area) < 13.16 and large: log2(area) > 13.16).

**Figure 7.** Histograms of occupied pixel area at *log*2-scale for all annotated objects by object category, divided into three groups for each category: small, medium and large according to Microsoft COCO variants (small: log2(area) < 10, medium: 10 < log2(area) < 13.16 and large: log2(area) > 13.16). The vertical colored lines represent the following values: the red line—represents the mean of the distribution, the yellow line represents the threshold for small objects and the green vertical line delineates the threshold for large objects. In each histogram, respectively, entries to the left of the yellow line represent the small objects group, entries in between the yellow and the green line show the medium-sized objects group and those to the right of the green line depict the large objects group.

**Table 4.** The table shows the number of images and annotations in the ABOships dataset for every object category, along with their overall percentages.

#### **4. Results**

#### *4.1. Evaluation Criteria*

To evaluate the performance of different object detection algorithms on specific datasets, one can employ various quantitative indicators. One of the most popular measures in object detection is the *IoU* (Intersection of Union ), which defines the extent of overlap of two bounding boxes as the intersection between the area of the predicted bounding box *B<sup>p</sup>* and the area of the ground truth bounding box *Bgt*, over their union [21]:

$$IoU = \frac{|B\_p \cap B\_{gt}|}{|B\_p \cup B\_{gt}|} \tag{1}$$

Given an overlap threshold *t*, one can estimate whether a predicted bounding box belongs to the background (*IoU* < *threshold*) or to the given classification system (*IoU* > *threshold*). With this measure, one can proceed to assess the average precision (*AP*) by calculating the precision and recall. The precision reflects the capability of a given detector to identify relevant objects and it is calculated as the proportion of detected bounding-boxes, correctly identified, over the total number of detected boxes. The recall reflects the capability of a detector to identify relevant cases and it is calculated as the proportion of correct positive predictions to all ground truth bounding boxes. Based on these two metrics one can draw a precision-recall curve, which encloses an area representing the average precision. However, in a majority of cases, this curve is highly irregular (zigzag pattern) making it challenging to estimate the area under it, i.e., the *AP*. To address this, one can approach it as an interpolation problem, either as an 11-point interpolation or an all-point interpolation [21].

The 11-point interpolation averages the maximum values of precision over 11 recall levels that are uniformly distributed [21], as depicted below:

$$AP\_{11} = \sum\_{\mathcal{R} \in \{0, 0.1, \dots, 0.9, 1\}} P\_l(\mathcal{R})\_\prime \tag{2}$$

with

$$P\_i(\mathcal{R}) = \max\_{\mathcal{R}^\* \mid \mathcal{R}^\* \ge \mathcal{R}} P\_i(\mathcal{R}^\*). \tag{3}$$

*AP*<sup>11</sup> is calculated using the maximum precision *Pi*(*R*), with a recall greater than *R*.

#### *4.2. Baseline Detection*

To explore the performance of CNN-based object detectors on our dataset, we focused on prevalent detectors: one-stage (SSD [18] and EfficientDet [22]) and two-stage detectors (Faster R-CNN [17] and R-FCN [23]). The detectors were previously trained on the Microsoft COCO object detection dataset, which comprises a number of 91 object categories. The training dataset contains a number of 3146 images of marine vessels. We investigated the performance of different feature extractors in the aforementioned detectors. We collect maritime vessel detection results based on SSD over different feature extractors (ResNet101, MobileNet v1, MobileNet v2). Moreover, we evaluate the performance of a new stateof-the-art detector, EfficientDet, on our dataset, which used EfficientNet D1 as feature extractor. We also evaluated two-stage detectors: Faster R-CNN and RFCN with different feature extractors. Combining all proposed detectors with the feature extractors, a total of 8 algorithms were investigated. All information regarding the specific configuration of these detectors can be found in [24].

We estimated the performance of these algorithms in detecting maritime vessels, so we excluded seamark and miscellaneous labels from our experiments and focused on detecting vessels. Moreover, we chose images with an occupied pixel area larger than 16<sup>2</sup> pixels. Based on these experiments, we attained Table 5.

Our experiments indicated that the object size impacts the detection accuracy. To corroborate this observation, we divided all vessel labels (with an occupied pixel area larger than 16<sup>2</sup> pixels) in our datasets into three categories, based on Microsoft COCO challenge's

variants: small (16<sup>2</sup> < area < 32<sup>2</sup> ), medium (32<sup>2</sup> < area < 96<sup>2</sup> ) and large (area > 96<sup>2</sup> ). Out of the annotated vessels with an occupied pixel area larger than 16<sup>2</sup> pixels in our dataset, 30.25% of the annotated vessels are small, 49.37% are medium and 20.37% are large.

Analyzing the results from our experiments, we observe that detection accuracy decreases with object size. The *AP* for best-performing detector on the ABOships dataset (Faster R-CNN with Inception ResNet v2 as feature extractor) with a registered *AP* of 35.18% more than doubles in size from small (*AP<sup>S</sup>* = 23.16%) to large objects (*AP<sup>L</sup>* = 46.84%). The second best detector on the whole dataset (EfficientDet with EfficientNet as feature extractor) however had the best performance on the large-objects category, with an *AP<sup>L</sup>* = 55.48%. In general, detecting small objects turns out to be more difficult than larger objects given that there is less information associated with a smaller occupied pixel area. For medium-sized objects, the best performance is attained by SSD with ResNet101 as feature extractor (*AP<sup>M</sup>* = 31.18%). For small objects, the best-performing detector, Faster R-CNN with Inception ResNet v2, outperforms the other detectors with a registered *AP<sup>S</sup>* = 23.16%. Among the SSD configurations, best performing, in general, was the one having ResNet101 as feature extractor.

**Table 5.** Average Precision (AP) (in %) of the proposed CNN-based detectors on ABOships dataset, with different feature extractors and object sizes, for all objects with an occupied pixel area > 16<sup>2</sup> pixels.

#### **5. Qualitative Results**

Figure 8 illustrates an example of detection results for the proposed methods, selecting for each the combination of feature extractor that scored the highest AP in each category. We can notice in Figure 8 that Faster R-CNN with a Inception-ResNet-v2 feature extractor (a) and R-FCN with a ResNet101 feature extractor (c) provide detected regions registering high scores ranging from 0.91 to 0.99. The other two detectors in Figure 8, EfficientDet with EfficientNet as feature extractor (b) and SSD with ResNet101 as feature extractor (d), register satisfying results registering with scores ranging from 0.55 to 0.67.

**Figure 8.** Qualitative detection results for the ABOships dataset on (**a**) Faster R-CNN and Inception-ResNet-v2 as feature extractor, (**b**) EfficientDet with EfficientNet as feature extractor, (**c**) R-FCN with ResNet101 as feature extractor, and (**d**) SSD with ResNet101 as feature extractor. The ground truth bounding-boxes are shown as red rectangles. Predicted boxes by these methods are depicted as green bounding boxes. Each output box is associated with a class label and a score with a value in the interval [0, 1].

#### **6. Discussion**

Maritime vessel detection of inshore and offshore images is a topical issue in many areas, such as maritime surveillance and safety, marine and coastal area management, etc. Many of these fields require intricate management of disparate activities, which in practice often necessitate real-time monitoring. This implies, among other aspects, realtime detection of inshore and offshore ships. However, in their majority, ship detection studies and methodology are mostly concerned with either satellite or radar imagery, which can prove to be unreliable in a real-time setting. For this very reason, algorithms, and specifically CNNs, employed on waterborne imagery are especially beneficial either on their own, or in fusion architectures.

Traditional ship detection methods using either background separation or histograms of oriented gradients provide satisfactory results under favorable sea conditions. However, the complexity of the marine environment, including challenging environmental factors (glare, fog, clouds, high waves, rain etc.), renders the extraction of low-level features unreliable. Recent studies involving CNNs address this issue, but deep learning requires domain-specific datasets to produce satisfactory performance. However, public datasets specifically designed for maritime vessel detection are scarce to this day [1]. We discuss this in more detail in Section 2.

Performing exploratory analysis on our dataset, in comparison with other recent maritime object detection datasets (Singapore Maritime Dataset [15], SeaShips [3], MC-Ships [19]), there are a few aspects that emerge that we discuss as follows. Comparing our dataset to the Singapore Maritime Dataset, one can notice (from Table 3) that ABOships registers a higher number of ship types (9 vs. 6). However, considering the number of annotations per image, the Singapore dataset registers almost 3 times more annotations on average per image (11.05 vs. 4.2). The SeaShips dataset consists of 31, 455 images, more than 3 times the image total of our dataset, but ABOships provides more annotations than the former, with a greater average number of annotations per image (4.2 vs. 1.2). SeaShips consists mostly of images with one annotation per image. MCShips provides a number

of 13 ship categories (vs. 9 ship categories in ABOships), but only offers just over 26*K* annotations, with an average of 1.8 annotations per image, see Table 3. We note that our dataset annotations comprise also seamarks and miscellaneous floaters in addition to the 9 ship types.

We tested our relabelling software application on the Singapore Maritime Dataset, as suggested by our reviewers, and the tracker was able to consistently map object labels from one frame to another correctly (without drifting from the object of interest to other objects), which did not always occur when we performed the tracking on the ABOships dataset. There are a few aspects that can influence the tracker's performance and those most probably affected its performance on the ABOships dataset. First, the videos included in the Singapore Maritime dataset have a higher frame rate (30 FPS), double than those in our dataset (15 FPS). Moreover, the videos from the Onshore dataset (one part of the Singapore Maritime Dataset) have higher resolution. Videos in the Onshore dataset do not have a high density of annotations per video. Furthermore, the environment present in the images of our dataset is far more complex, including urban landscapes and complicated background, especially in the port area.

#### **7. Conclusions**

This paper provides a solution for addressing the annotation inconsistencies appeared as a consequence of manual labeling of images, using the CSRT tracker [20]. We build traces of the images in the videos they originated from and use the CSRT tracker to traverse these videos in both directions and identify the possible inconsistencies. After this step, we employed a second round of labeling and obtained a set of 41, 967 carefully annotated objects, of which 9 types of maritime vessels (boat, miscboat, cargoship, passengership, militaryship, motorboat, ferry, cruiseship, sailboat), miscellaneous floaters and seamarks.

We ensured the dataset consists of images taking into account the following factors: background variation, atmospheric conditions, illumination, visible proportion, occlusion and scale variation. We performed a comparison of the out-of-the-box performances of four state-of-the-art CNN-based detectors (Faster R-CNN [17], R-FCN [23], SSD [18] and EfficientDet [22]). These detectors were previously trained on the Microsoft COCO dataset. We assess the performance of these detectors based on feature extractor and object size. Our experiments show that Faster R-CNN with Inception-Resnet v2 outperforms the other algorithms for objects with an occupied pixel area > 16<sup>2</sup> pixels, except in the large object category where EfficientDet registers the best performance with an *AP* = 55.48%.

For future research, we plan to investigate different types of errors in the manual labelling, for cases where the labels still have inconsistencies, such as: fine-grained recognition (which renders it more difficult for human even to detect objects even when they are in plain view [25], class unawareness (some annotators become unaware of certain classes as ground truth options) and insufficient training data (not enough training data for the annotators).

Moreover, we plan to investigate in more detail the detection of small and very small objects, including those with an occupied pixel area < 16<sup>2</sup> pixels. Furthermore, distinguishing between different vessel types in our datasets will be an essential focus as the next steps in our experiments. In order to do this, we plan to exploit transfer learning both in the form of heterogeneous transfer learning, but also homogeneous domain adaptation.

To further our research, we will employ maritime vessel tracking detectors on the original videos captured in the Finnish Archipelago and examine the impact on autonomous navigation and navigational safety.

**Author Contributions:** V.S. and J.L. planned video capture and collection in the Finnish Archipelago. B.I. and V.S. planned the annotation process and wrote the annotation requirements. L.Z. and V.S. supervised and participated in the annotation process. V.S. implemented the relabelling algorithm. B.I. planned the experiments on the relabelled dataset and supervised their implementation. L.Z. wrote the software for the evaluation of the algorithms on the datasets. V.S. wrote the software for

AP calculations. All authors contributed to the interpretation of results. B.I. wrote the following sections and subsections: Introduction, Conclusion, Experimental Results, Dataset Statistics, Dataset Diversity, Annotation. L.Z. and V.S. wrote the Related Work section. Valentin Soloviev wrote the following subsections in Materials and Methods: Dataset Design, Relabelling Algorithm. The annotation subsection was written by V.S. and B.I. B.I. planned the manuscript writing, and revised the final writing of each section. B.I. and J.L. supervised the evaluation of the algorithms, AP calculation. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Data Availability Statement:** The data presented in this study are available on request from the corresponding author. The data are not publicly available due to being in the process of publishing, it is planned to be published at: https://www.fairdata.fi/en/ (accessed on 4 February 2021). For reviewers we can provide a separate package with data and any necessary code in the meantime.

**Acknowledgments:** The annotation of the ABOships dataset was completed with the help of the following persons: Sabina Bäck, Imran Shahid, Joel Sjöberg and Alina Torbunova.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

## *Article* **A Co-Operative Autonomous Offshore System for Target Detection Using Multi-Sensor Technology**

#### **Jose Villa 1,\* , Jussi Aaltonen <sup>1</sup> , Sauli Virta <sup>2</sup> and Kari T. Koskinen <sup>1</sup>**

Received: 16 November 2020; Accepted: 11 December 2020; Published: 16 December 2020

**Abstract:** This article studies the design, modeling, and implementation challenges for a target detection algorithm using multi-sensor technology of a co-operative autonomous offshore system, formed by an unmanned surface vehicle (USV) and an autonomous underwater vehicle (AUV). First, the study develops an accurate mathematical model of the USV to be included as a simulation environment for testing the guidance, navigation, and control (GNC) algorithm. Then, a guidance system is addressed based on an underwater coverage path for the AUV, which uses a mechanical imaging sonar as the primary AUV perception sensor and ultra-short baseline (USBL) as a positioning system. Once the target is detected, the AUV sends its location to the USV, which creates a straight-line for path following with obstacle avoidance capabilities, using a LiDAR as the main USV perception sensor. This communication in the co-operative autonomous offshore system includes a decentralized Robot Operating System (ROS) framework with a master node at each vehicle. Additionally, each vehicle uses a modular approach for the GNC architecture, including target detection, path-following, and guidance control modules. Finally, implementation challenges in a field test scenario involving both AUV and USV are addressed to validate the target detection algorithm.

**Keywords:** target detection; co-operative; autonomous; multi-robot; USV; AUV

### **1. Introduction**

In recent years, the use of autonomous offshore vehicles, which includes autonomous underwater vehicles (AUVs) and unmanned surface vehicles (USVs), for marine interventions has attracted increasing interest from research scientists, maritime industries, and the military. These interventions include several activities such as offshore surveillance, offshore target detection, seabed explorations, or search and rescue (SAR) missions. Additionally, the use of multi-robot platforms can improve the performance in these activities, as they can include above and below-water characterization. Regarding a multi-robot platform, Vasilijevi´c et al. [1] presented the co-operative robotic system consisting of an AUV and a USV for ocean sampling and environmental monitoring. In [2], the study used a heterogeneous collaborative system of above, surface, and underwater robots to obtain a multi-domain awareness on a floating target. The heterogeneous system consists of a USV, an AUV, and an unmanned aerial vehicle (UAV). Additionally, Gu et al. [3] presented a homogeneous study, where a guidance and control law design method for coordinated path following of networked under-actuated robotic USVs under directed communication links. In [4], the control scenario simulated a homogeneous AUV fleet to study formation tracking control and collision-obstacle avoidance.

To accomplish the target detection in the offshore environment, the availability of accurate USV and AUV mathematical models is crucial for simulation study purposes, controller design, and development. The theoretical six-degrees-of-freedom (DOFs) dynamic model [5], based on nonlinear equations of

motion, can be used for the design and modeling of the AUV. Equally, the USV can use the same dynamic model of the AUV but with reduced order for the three DOFs horizontal plane control (surge, sway, and yaw motions). Several tools can help to obtain the coefficients of the dynamic model equations and the necessary transfer functions of each vehicle. These tools can include the parameter estimation from MATLAB-Simulink [6], and the system identification (SI) [7,8], introduced to develop the mathematical model using field test data. In [9], SI of the maneuvering data determined the hydrodynamic coefficients of a USV. Also, the mathematical model of the USV includes the propulsion and power system. Commonly, the rudder and propeller, or waterjet propulsion systems provide the heading and the speed control of most existing USVs. In [10], a twin waterjet propelled USV was modeled based on SI, but it neglects the calculation for the dynamics of the propulsion system.

Target detection in offshore environments is a fundamental activity that combines different perception sensors. Numerous studies use passive (stereo cameras) or active (LiDAR or radar) perception methods to obtain situational awareness of a USV. Nonetheless, most of the obstacle detection methods rely on depth measurements, in which LiDAR sensors are the most reliable method of obtaining depth data. Correspondingly, sonar devices are still the most convenient option for collecting data on underwater environments. Mechanical imaging sonar, multibeam, profiler, or sidescan are some of the main sonar imaging and ranging devices. For the target detection with sonar devices, how detectable is a target is mainly dependent on the physical characteristics of the target and acoustic signal. Some studies use sonar devices for target detection capabilities, as in [11], where a profiler sonar was adopted for obstacle detection. According to [12], a method for underwater obstacle detection (standard buoy) was developed using forward-looking sonar and a probabilistic local occupancy grid.

Correct localization and navigation are crucial to ensure the accuracy of the gathered data for all these applications. Above the water surface, most of the autonomous systems rely on radio or global positioning and spread-spectrum communications, as a GPS-compass installed in the USV platform. However, those signals propagate only in short distances in an underwater scenario, where acoustic-based systems perform better. Regarding underwater navigation, the three fundamental methods are deadreckoning (DR) and inertial navigation systems (INS), acoustic navigation, and geophysical navigation techniques [13]. These navigation methods require specific survey and navigation sensors installed in the AUV. The Girona 500 [14] is an example of AUV that performs the traditional dead-reckoning navigation utilizing a doppler velocity log (DVL) and a solid-state attitude and heading reference system (AHRS). Also, the absolute position can be obtained through a GPS when the vehicle is on the surface and using an ultra-short baseline (USBL) while underwater. The high-accuracy USBL system allows the localization of the AUV and the communication between the vehicle and the surface unit. In [15], the study provided a navigation algorithm for an underwater vehicle with a Kalman filter to estimate the error state via measurement residuals from aiding sensors.  In acoustic navigation techniques, acoustic transponders and modems perform localization by measuring the time-of-flight of signals from acoustic beacons or modems. USBL navigation allows an AUV to localize itself relative to a USV, and it provides an efficient and reliable acoustic communication network [16]. In [17], the study presented the design and implementation of an USBL-aided navigation approach for an AUV in a two-parallel extended Kalman filter (EKF). It also includes the measurements provided by a DVL, a Visual Odometer, an inertial measurement unit (IMU), a pressure sensor, and a GPS.

Safe and adequate control of the offshore vehicles depends notably on proper guidance, navigation, and control (GNC) systems. This study adopts a path-following as the guidance system for both offshore platforms. The path-following approach is closer to practical engineering, and it is easier to implement than trajectory tracking. A generally used method for path-following in autonomous vehicles is the named line-of-sight (LOS) guidance. LOS guidance is classified as a three-point guidance scheme, involving a commonly stationary reference point along with the interceptor and the target [5]. In [18], the study developed a guidance-based algorithm for path-following using the LOS algorithm

in offshore operations. Additionally, in [10], a path-following with obstacle avoidance based on the safety boundary box approach was implemented in a USV with a LOS-based guidance system.

Due to the co-operative offshore system in this study, it becomes necessary to fuse information obtained from the individual vehicles. Robot Operating System (ROS) has been an effective tool when working with multi-robot systems. This tool is a flexible framework for writing robot software and provides the tools to acquire sensors' data, process it, and generate the necessary response for the vehicle actuators [19]. Multi-robot systems can either be centralized with a ROS master node at the ground control station (GCS) or decentralized with each autonomous vehicle (AV) running an independent ROS master. In the case of the decentralized control techniques, they are more flexible, profitable, and generally reduce the communication network requirements compared with centralized control [20]. However, they are also more challenging due to obstacles, uncertainties, and communication constraints, such as noises, delays, dropouts, or failures. In this case, the multi-master approach provides a solution where each vehicle keeps its own ROS master and also exchange the necessary information with other components of the multi-robot system. In [21], they proposed a package that efficiently developed multi-master architectures.

In the presented manuscript, the mathematical model of the USV consists of the simplified three DOFs dynamic model [5], where their parameters are obtained from field test data using the parameter estimation tool. Additionally, the waterjet model has been included in the mathematical model of the USV using data from the manufacturer and transfer functions based on SI. The AUV platform considered in this study does not incorporate a DVL, neglecting the velocity feedback of the vehicle. However, the installed USBL provides an absolute position and a communication link between the USV and the AUV. Thus, the AUV platform includes a basic setup for underwater localization, but it is not able to precisely locate the vehicle underwater. The path-following algorithm uses the LOS approach for heading control to simplify the guidance control of the AUV, keeping a constant depth and constant surge speed. The target detection algorithm uses a modular ROS architecture to provide a computationally cheap and simple implementation in both offshore platforms. Furthermore, the offshore system includes two different perception sensors based on the same target detection algorithm. Finally, a multi-master architecture is in charge of the interaction between the AUV and USV, providing an easy plug-and-play solution for the multi-robot system.

In this work, a model-based GNC architecture for a co-operative autonomous offshore system is proposed for target detection using multi-sensor technology. In Section 2, the USV modeling and simulation are presented using the parameter estimation tool to define the waterjet and USV maneuvering model. Furthermore, this section includes an overview of the USV and AUV platforms. Then, in Section 3, the GNC system for the co-operative tasks is included using the LOS-based guidance system for control. The target detection algorithm is developed using a mechanical imaging sonar at the AUV and a LiDAR at the USV as the primary perception sensor for underwater and surface inspection, respectively. Finally, in Section 4, the implementation of a GNC architecture is described as modular and multilayer for the multi-robot system. A control scenario in a field test is shown in this section to validate the proposed target detection algorithm.

#### **2. Modeling and Simulation for the Offshore Vehicles**

The co-operative autonomous offshore system consists of two different vehicles: a USV and an AUV. This section gives an overview of both subsystems, and it describes the simulation model of the USV, which provides the capability to develop the GNC algorithms.

#### *2.1. Overview of Under-Actuated USV*

This article uses an under-actuated USV as the primary vehicle in the co-operative autonomous offshore system. The USV is an aluminum hull with a thrust vectoring waterjet propulsion system, which provides optimal maneuverability using a twin waterjet configuration. Figure 1 shows a simplified model of the vehicle, where the port and starboard (STDB) waterjets produce the necessary thrust forces to move forward, backward, sideways or performing turns. Additionally, Figure 1 includes the position and orientation of the USV in the North-East-Down (NED) coordinate system. The NED coordinate system is related to planar Cartesian coordinates, so a coordinate transformation is performed from the GPS-compass output to get the USV's absolute position. This transformation is between longitude and latitude (*l*, *µ*) from the world geodetic system 84 (WGS84) coordinate system and ETRS-TM35FIN [22], which displays the NED position (*x*USV, *y*USV). The Euler angles provide the USV heading or yaw angle *ψ*. The motion of the USV has three DOFs, which are surge, sway, and yaw (linear (*u*, *v*), and angular *r* velocities) while ignoring roll, pitch, and heave motions.

**Figure 1.** Simplified model of the unmanned surface vehicle (USV) using the North-East-Down (NED) coordinate system. USV motion is described by surge *u* (linear longitudinal motion), sway *v* (linear transverse), and yaw motion *r* (turning rotation about its z-axis).

#### *2.2. USV Modeling*

The development of an adequate maneuvering model will simplify the GNC algorithms design and simulation. The three DOFs horizontal plane model for maneuvering of a USV consists of the rigid-body kinetics [5]

$$\mathbf{M}\dot{\boldsymbol{\nu}} + \mathbf{C}(\boldsymbol{\nu})\boldsymbol{\nu} + \mathbf{D}(\boldsymbol{\nu})\boldsymbol{\nu} = \boldsymbol{\pi} + \boldsymbol{\pi}\_{\text{wind}} + \boldsymbol{\pi}\_{\text{wave}} \tag{1}$$

where *ν* = [*u*, *v*,*r*] *T* is the velocity vector composed of surge, sway and yaw. *τ* = [*τ*u, 0, *τ*r] is the vector forces and moments generated by twin waterjet configuration, while *τ*wind and *τ*wave are the environmental forces. *M*, *C*(*ν*), and *D*(*ν*) are the mass, Coriolis and damping matrices, respectively, where *M* and *C*(*ν*) combine added and rigid-body terms. The mass matrix *M* is defined by

$$\mathbf{M} = \mathbf{M\_{RB}} + \mathbf{M\_{A}} = \begin{bmatrix} m - \mathbf{X\_{il}} & \mathbf{0} & \mathbf{0} \\ \mathbf{0} & m - \mathbf{Y\_{\vartheta}} & m\mathbf{x\_{g}} - \mathbf{Y\_{\vartheta}} \\ \mathbf{0} & m\mathbf{x\_{g}} - \mathbf{Y\_{\vartheta}} & \mathbf{I\_{z}} - \mathbf{N\_{\vartheta}} \end{bmatrix},\tag{2}$$

where *m* is the mass of the vehicle, *I*<sup>z</sup> is the moment of inertia about *z*<sup>b</sup> axis, *r b* <sup>g</sup> = - *x*g, *y*g, *z*<sup>g</sup> | is the vector from origin *o*<sup>b</sup> to centre of gravity CG, and *Xu*˙ , *Yv*˙ , *Yr*˙ , and *Nr*˙ represent hydrodynamic added mass. The moment of inertia *I*<sup>z</sup> at the pivot point has been estimated based on the calculation of the moments of inertia in the rear *I*z,rear and front *I*z,front of the USV. These moments of inertia are defined by

$$I\_{\rm z,near} = m\_{\rm pt} l\_{\rm pt}^2 + \left(\frac{1}{3} \, m\_{\rm hull} \, c\_{\rm g} \right) l\_{\rm pivot}^2 \tag{3}$$

$$I\_{\rm z,front} = \frac{1}{3} \,\mu\_{\rm hull} \left(1 - c\_{\rm g}\right) \times \left(l\_{\rm USV} - l\_{\rm pitot}\right)^2 \,\text{\,\,\,}\tag{4}$$

where *m*pt is the estimated powertrain mass (engines, waterjets, fuel, etc.), *l*pt is the estimated location of the powertrain mass, *m*hull is the hull weight without powertrain mass, *c*<sup>g</sup> is the relative center of mass point having one as the front of the USV, *l*pivot is the pivot point location, *κ* is a scaling factor as the mass is not evenly distributed from the pivot point to the front of the USV, and *l*USV is the length of the USV. The total moment of inertia *I*<sup>z</sup> is defined by

$$I\_\mathbf{z} = \left(I\_{\mathbf{z,near}} + I\_{\mathbf{z,front}}\right) I\_{\mathbf{Corr}} \tag{5}$$

where *I*cor is the tuning factor for the moment of inertia.

The Coriolis-centripetal matrix *C*(*ν*) can always be parameterized such that *C*(*ν*) = *C* | (*ν*) [23]. However, linearization of the Coriolis and centripetal forces *C*RB(*ν*) and *C*A(*ν*) about zero angular velocity (*p* = *q* = *r* = 0) implies that the Coriolis and centripetal terms can be removed from the above expressions, that is *C*RB(*ν*) = *C*A(*ν*) = 0 [24]. Additionally, the mathematical model is simplified to take into account only surge and yaw motions, so Coriolis and centripetal terms have been removed at the three DOFs dynamic model in this study.

The different damping terms contribute to linear and quadratic damping [5]. Nonetheless, it is generally difficult to distinguish these effects. The total hydrodynamic damping matrix *D*(*νr*) is the sum of the linear part *D*lin and the nonlinear part *D*nlin(*νr*) such that

$$D(\nu\_r) = D\_{\text{lin}} + D\_{\text{nlin}}(\nu\_r) \tag{6}$$

where *D*lin is the linear damping matrix produced by potential damping and possible skin friction, and *D*nlin(*νr*) is the nonlinear damping matrix as a result of the quadratic damping and higher-order terms, defined by

$$\mathcal{D}\_{\text{lin}} = \begin{bmatrix} -X\_{\mu} & 0 & 0 \\ 0 & -Y\_{\upsilon} & -Y\_{r} \\ 0 & -Y\_{r} & -N\_{r} \end{bmatrix} \tag{7}$$

$$\mathcal{D}\_{\text{nlin}}(\nu\_r) = \begin{bmatrix} -X\_{|u|u} & 0 & 0 \\ 0 & -Y\_{|v|v} & 0 \\ 0 & 0 & -N\_{|r|r} \end{bmatrix} \left| \nu\_r \right| \,. \tag{8}$$

The USV used in this study includes the AJ245 waterjet units [25]. The nozzle position *P*nozzle varies the direction of the jet flow, which generates the force needed for turning. Thus, the total thrust force *F*total combines the engine rpm of the waterjet *n*rpm and *P*nozzle. The variable *n*rpm is directly gathered from the waterjet engine, and *P*nozzle is a variable from −10,000 to 10,000, with 0 as the neutral position and equal to forward motion. Table 1 shows the data obtained from the manufacturer Alamarin-Jet Oy for these waterjet units at a specific operating point. This operating point is selected at 1800 rpm, nozzle in the neutral position, and bucket in the full up position.

**Table 1.** Data obtained from manufacturer for an operating point of a single AJ245 waterjet unit.

The thrust forces and torques for the mathematical model of the USV are defined according to the manufacturer's data and an affinity law. Thus, a two-dimensional (2D) lookup table can include the relation between the shaft rotational speed of the waterjet engine *N* with the thrust force per waterjet *F*. The affinity law used to obtain the thrust force at the waterjet units is defined by

$$\frac{F\_1}{F\_2} = \left(\frac{N\_1}{N\_2}\right)^2. \tag{9}$$

Figure 2 shows the results for the affinity law with the manufacturer's data for a waterjet engine from 600 to 2400 rpm, which match the operational engine speeds of this study.

**Figure 2.** Thrust force *F* generated by the waterjet propulsion system depending on the shaft rotational speed *N*.

In the mathematical model, a 2D lookup table provides the engine rpm and the surge speed of the USV as inputs, and the total thrust generated by the waterjet unit as output. Also, a one-dimensional (1D) lookup table *f*(*Joyu*) obtains the engine rpm depending on the joystick input for surge motion, and a second-order transfer function adds the waterjet dynamics of the engine rpm into the mathematical model. This transfer function is obtained using the SI tool from MATLAB and the field test data of the USV. Thus, the engine rpm is calculated based on the combination of the 1D lookup table and the engine rpm transfer function, defined by

$$m\_{\rm rpm}(s) = \frac{0.317s^2 + 2.793s + 1.828}{s^2 + 3.499s + 1.828} f(log\_u). \tag{10}$$

In the case of the heading motion of the USV, the total efficiency *η*nozzle for the thrust force depends on the nozzle position (which refers to the angle of the waterjet thrust force *α*nozzle). According to the waterjet manufacturer, if the nozzle position is deviated to a maximum nozzle angle *η*nozzle = ±25◦ (related to *P*nozzle = ±10,000) , efficiency drops exponentially to 30–40% of the maximum (center). The exponential function is obtained using the general exponential model.

$$
\eta\_{\text{nozzle}}(P\_{\text{nozzle}}) = a \exp(b \, P\_{\text{nozzle}}),
\tag{11}
$$

where *<sup>a</sup>* <sup>=</sup> 1 and *<sup>b</sup>* <sup>=</sup> <sup>−</sup>9.163 <sup>×</sup> <sup>10</sup>−<sup>5</sup> .

Similarly to the dynamics of the waterjet calculation for the engine rpm, the nozzle position includes a 1D lookup table *f*(*Joyr*) and a first-order transfer function. This transfer function is obtained also from the SI tool from MATLAB based on field test data. The nozzle position of each waterjet is defined by

$$P\_{\text{nozzle}}(s) = \frac{-\exp(-0.25s)}{0.1s + 1} f(Jy\_r). \tag{12}$$

Regarding the behavior of the second-order transfer functions for both engine rpm and nozzle position, Figure 3 shows the comparison between the SI tool transfer function and field test data for both *n*rpm and *P*nozzle variables.

**Figure 3.** Comparison between the USV field test data and the system identification (SI) transfer functions: (**a**) Engine rpm *n*rpm. (**b**) Nozzle position *P*nozzle.

Additionally, the parameters for the 1D Lookup table are obtained from field test data and are presented in Table 2.

**Table 2.** 1D Lookup Table parameters.

Finally, the vector *τ* = [*τ*u, 0, *τ*r], which represents the forces and moments generated by the two waterjets, is defined by

$$\begin{cases} \tau\_{\rm u} = (F\_{\rm POKT} + F\_{\rm STDB}) \eta\_{\rm nozzle} \\ \tau\_{\rm r} = l\_{\rm pivot} \sin(a\_{\rm nozzle}) (F\_{\rm POKT} + F\_{\rm STDB}) \eta\_{\rm nozzle} \end{cases} \tag{13}$$

Figure 4 shows the schematic with all the necessary functions for the USV dynamic model, from the joystick controller input to the vehicle's position output. The waterjet model includes the 1D lookup table to translate between joystick commands to rpm, the second-order transfer function, and the 2D lookup table related to the thrust force of each waterjet unit. Furthermore, it also includes the 1D lookup table to translate between joystick commands to the nozzle position, the first-order transfer function, the thrust force efficiency depending on the nozzle position, and the calculation of the total torque. Both thrust force *τ*<sup>u</sup> and torque *τ*<sup>r</sup> are the inputs in the mathematical model of the USV based on the three DOFs dynamic model. The position and orientation of the USV are performed by integrating the velocity vector *ν*.

**Figure 4.** Schematic of the mathematical model of the USV including both waterjet propulsion system and USV dynamic models.

#### *2.3. USV Model-Validation Using Parameter Estimation*

The matrices *M* and *D*(*ν*) of the three DOFs Dynamic model are estimated with the parameter estimation tool from MATLAB-Simulink. The matrices are defined in the Simulink model by creating the matrices from input values. Then, the MATLAB-Simulink tool can estimate the individual coefficients of the dynamic matrices.

There are two different parameter estimation runs related to surge and yaw motion. Table 3 shows the constant values shared in both experiments, while Table 4 shows the coefficients obtained from the parameter estimation tool with their results. Only surge and yaw motion coefficients, *Xu*, *Xu*˙ , *<sup>X</sup>*|*u*|*<sup>u</sup>* and *N<sup>r</sup>* , *Nr*˙ , *<sup>N</sup>*|*r*|*<sup>r</sup>* respectively, have been considered and estimated in this study, as the mathematical model focuses in these two USV motions.

**Table 3.** Principal characteristics of the under-actuated USV.

**Table 4.** Parameter estimation results for the surge and yaw motion coefficients.

Figure 5 shows the comparison between the field tests, which include raw and filtered USV linear and angular velocity, the three DOFs dynamic model with the coefficients obtained from the parameter estimation, and the SI results from [10], for the joystick controller input shown in Figure 3. As shown

in both linear and angular velocities results, the parameter estimation results improve the previous SI approach, giving an accurate output of the USV maneuvering compared to the field test results.

**Figure 5.** Comparison plot between SI tool, parameter estimation (PE) app, and field test data: (**a**) Surge motion. (**b**) Heading motion.

#### *2.4. Overview of the AUV*

This article uses a high configurable AUV platform for different scientific instrumentation. This vehicle contains basic instrumentation and sensors for localization and target detection, including a USBL and a depth sensor for underwater localization and navigation, an AHRS from the flight control for the navigation of the AUV, and a mechanical imaging sonar (Tritech Micron [26]) as main underwater perception sensor.

Figure 6a shows a simplified model of the AUV. This AUV uses a six-thruster configuration to provide thrust forces when moving in the surge, sway, heave motions, or performing turns. Also, the position and velocities of the AUV are illustrated in Figure 6a. The general AUV motion in six DOFs is modeled by using the NED local coordinate system. AUV position and velocities are considered with the following vectors

$$\mathfrak{h} = \left[ \mathbf{N}, \mathbf{E}, \mathbf{D}, \boldsymbol{\Phi}, \boldsymbol{\Phi}, \boldsymbol{\Psi} \right]^{\top}, \boldsymbol{\nu} = \left[ \boldsymbol{u}, \boldsymbol{v}, \boldsymbol{w}, \boldsymbol{p}, \boldsymbol{q}, \boldsymbol{r} \right]^{\top}, \tag{14}$$

where *N*, *E*, *D* denote the NED positions in Earth-fixed coordinates, *φ*, *θ*, *ψ* are the Euler angles, *u*, *v*, *w* are the body-fixed linear velocities, and *p*, *q*,*r* are the body-fixed angular velocities [5].

The design and modeling of the AUV should be studied using a theoretical six DOFs dynamic model [27]. However, due to the lack of instrumentation, it is not possible to obtain accurate navigation data. Thus, the AUV is not fully simulated, and just simple control commands are established for navigation. Once that navigation data is available, it is possible to use the same approach as the USV mathematical model to obtain the six DOFs dynamic model, using the parameter estimation or SI tools based on field test data. Regarding the control of the AUV, thrusters are located as it is shown in Figure 6b, where thrusters *T*1, *T*2, *T*3, and *T*<sup>4</sup> effects in surge, sway, and yawing, and thrusters *T*<sup>5</sup> and *T*<sup>6</sup> effects in heave and rolling motions.

**Figure 6.** Six-thruster configuration in the AUV: (**a**) Simplified model of the considered vehicle using the NED coordinate system. (**b**) Thrust forces with their direction for each thruster.

#### **3. Gnc System for the Co-Operative Tasks**

This study has the target detection and the guidance algorithms as main modules of the GNC architecture of the offshore multi-vehicle system. This section describes both of these algorithms for each platform and the description of the multi-vehicle guidance system.

#### *3.1. Target Detection System*

The mechanical imaging sonar installed at the AUV and the LiDAR at the USV are the primary perception sensors in the co-operative autonomous offshore system. The target detection algorithm includes the application in both perception sensors, depending on the position of the objects (underwater or over the water surface).

For the mechanical imaging sonar, the employed algorithm consists of analyzing the acoustic intensity at every bin to determine the presence of an underwater vehicle. The Tritech Micron sonar [26] has an operating frequency chirp centered on 700 kHz, a beamwidth of 35◦ vertical and 3◦ horizontal, a range from 0.3 to 75 m, a range resolution of approximately 7.5 mm, and a configurable mechanical resolution of 0.45◦ , 0.9◦ , 1.8◦ , and 3.6◦ . In this study, the maximum range used to detect an obstacle is 10 m, a forward field-of-view (FoV) of 90◦ , and a mechanical resolution of 1.8◦ . If the target is known a priori to be narrow, the imaging sonar can be configured with a lower resolution to detect the object.

Regarding the data obtained from the mechanical imaging sonar, it contains the heading of the beam *θ*scan, the location of the specific point in Cartesian coordinates *P*scan, and the intensity at every bin *I*scan. The dynamic range of the mechanical imaging sonar is 80 dB. Then, the dynamic range controls allow to adjust the position of a sampling window within the defined dynamic band range of the received signal, and it translates the intensity at every bin to an integer value ranging between 0 and 255.

After data acquisition from the mechanical imaging sonar, Algorithm 1 shows the post-processing steps for target detection. This algorithm includes the position of the highest intensity value for each bin in polar coordinates, filtering the data in the range of [0,1.5] meters to avoid possible noise from the AUV structure.

Algorithm 1 provides the post-processing of a single bin of a specific angle. An additional function forms an array of number of scans *n*scans, obtained from *θ*scan,min, *θ*scan,max, and *θ*scan,increment parameters of the mechanical imaging sonar to create the complete array of scans from the sonar. After gathering the scan array, the position of the targets needs to be calculated. The data from the perception sensors is obtained in the body-fixed reference frame (BODY), and it requires a translation into an absolute coordinate system. This translation is defined by

$$
\begin{bmatrix} \chi\_{\rm obs} \\ \mathcal{Y}\_{\rm obs} \end{bmatrix} = \mathbf{R}\_{\mathbf{z}}(\psi\_{\rm AV}) \begin{bmatrix} \chi\_{\rm scan} \\ \mathcal{Y}\_{\rm scan} \end{bmatrix} \tag{15}
$$

where *Rz*(*ψ*AV) is the rotation matrix around the z-axis using the heading angle *ψ*AV of the selected AV. This rotation matrix translates between the BODY and the East-North-Up (ENU) coordinate system. The rotation matrix *Rz*(*ψ*AV) in 2D is defined by

$$\mathbf{R}\_z(\psi\_{\rm AV}) = \begin{bmatrix} \cos(\psi\_{\rm AV}) & \sin(\psi\_{\rm AV}) \\ -\sin(\psi\_{\rm AV}) & \cos(\psi\_{\rm AV}) \end{bmatrix} \tag{16}$$

**Algorithm 1:** Post-processing of the mechanical imaging sonar data for target detection.

```
Input :Intensities Iscan, positions Pscan in Cartesian coordinates [X,Y], and current heading
       θscan value obtained from the mechanical imaging sonar.
```
**Output :**Position *micron* of the highest intensity value in polar coordinates.

	- /\* Remove data in the range from 0 to 1.5 m to avoid possible noise from the AUV structure. *nscan* equal to number of scans. \*/

After locating the obstacle by the mechanical imaging sonar in the ENU coordinate system, the target's origin position (*N*o, *E*o) is defined by

$$
\begin{bmatrix} N\_{\rm o} \\ E\_{\rm o} \end{bmatrix} = \begin{bmatrix} N\_{\rm AV} \\ E\_{\rm AV} \end{bmatrix} + \mathbf{R}\_{\rm x}(\gamma) \begin{bmatrix} \frac{\mathbf{x}\_{\rm obs,init} + \mathbf{x}\_{\rm obs,end}}{2} \\ \frac{\mathbf{y}\_{\rm obs,init} + \mathbf{y}\_{\rm obs,end}}{2} \end{bmatrix} \tag{17}
$$

where *Rx*(*γ*) is the rotation matrix around x-axis with *γ* = *pi* [rad]. This matrix is used to translate between ENU to NED coordinate system used for the offshore navigation. The *Rx*(*γ*) rotation matrix in 2D is defined by

$$\mathbf{R}\_{\mathbf{x}}(\gamma) = \begin{bmatrix} 1 & 0\\ 0 & \cos(\gamma) \end{bmatrix}. \tag{18}$$

Algorithm 2 includes the detected target localization for the perception sensor data array. This algorithm distinguishes between different targets depending on the consecutive elements in the data array, and the origin position of the targets is sent to the GNC algorithm to proceed with the autonomous navigation of the offshore system.

#### **Algorithm 2:** Localization of the detected targets.

**Input :***scan* data array in Cartesian coordinates and *RobotPose* (position and orientation). **Output :**Obstacle origin [*N*o, *E*o] calculated in absolute NED coordinates.

**5** create vector to distinguish between different obstacles;

```
9 for i = 1 to nobs do
```

```
10 calculate the obstacle origin [No(i), Eo(i)] in NED according to (17);
```

**11 end**

```
14 end
```
Figure 7 shows the steps from the scan data obtained from the mechanical imaging sonar in the BODY reference frame to the final origin position of the detected targets. Figure 7a shows the raw data from the mechanical imaging sonar. Then, Figure 7b shows the post-processing described in Algorithm 1. Finally, Figure 7c,d represents the origin position of the targets in NED coordinate system, with relative to origin [0,0] and absolute coordinates respectively.

**Figure 7.** Post-processing of the mechanical imaging sonar data in the target detection algorithm: (**a**) Scan data acquired from sonar. (**b**) Post-processing based on Algorithm 1. (**c**) Relative position in NED with origin as [0,0] and calculation of target's origin. (**d**) Absolute position in NED of the targets.

Regarding the USV platform, the SICK MRS1000 LiDAR [28] is the primary perception sensor. This LiDAR has four spread-out scan planes and a multi-echo analysis to be used in harsh environment applications, as it can avoid the noise produced by fog, rain, or dust. Also, this device has a 275◦ aperture angle, and a working range from 0.2 to 64 m. Thus, in case that the target is above the water surface, it can be detected by the LiDAR sensor.

The algorithm for target detection is similar to the described for the mechanical imaging sonar. The only difference is that the LiDAR contains four spread-out scan planes, acquiring three-dimensional (3D) scan data (see Figure 8a). The target detection algorithm is simplified by translating the received data to 2D by avoiding the z-axis from the sensor data (see Figure 8b). Figure 8c shows the maximum detection range and aperture angle with the scan data in the BODY reference frame. Finally, Figure 8d shows the origin's position of the targets in the NED coordinate system after applying Algorithm 2.

**Figure 8.** Post-processing of the LiDAR in the target detection algorithm: (**a**) LiDAR scan data in 3D. (**b**) LiDAR scan data in 2D. (**c**) Detection area of the USV in BODY including scan data. (**d**) Absolute position in NED of the targets.

The same procedure detects obstacles from the LiDAR for the path-following with the obstacle avoidance algorithm. After obtaining the origin position [*N*o, *E*o] from Algorithm 2, the obstacle avoidance algorithm can define a safety boundary box around the obstacle [10].

#### *3.2. Guidance System for Multi-Vehicle System*

The multi-vehicle system aims firstly to detect a target using the AUV in a specific offshore area, and after that, sends the location to the USV to do further exploration of the target. Thus, a pathfollowing algorithm is essential for both AUV and USV subsystems. This algorithm intends to reach every waypoint of a specific path independent of time. A commonly used method for path-following is the named LOS guidance, which is chosen as a reference trajectory in this study.

#### 3.2.1. Auv Guidance System

The heading control can use a LOS vector from the AUV position to the next waypoint, similar to [5]. The LOS path-following controller used in this study is the same as the one defined in [10]. However, the AUV movement includes a heave motion, which is avoided by keeping a constant depth for the path-following algorithm. This controller computes the course angle *ψ*<sup>d</sup> based on the path-tangential angle *χ*<sup>p</sup> and the velocity-path relative angle *χ*r. The lookahead-based steering can be implemented related to the heading controller applying the transformation defined as

$$
\psi\_{\rm d} = \chi\_{\rm P} + \chi\_{\rm r} - \beta\_{\rm \prime} \tag{19}
$$

where the variable sideslip (drift) angle *β* [5] has been omitted in this study to simplify the steering law. The velocity-path relative angle *χ*<sup>r</sup> establishes that the velocity has the direction facing a path location that is in a lookahead distance ∆(*t*) > 0 along of the direct projection [29]. The path-tangential angle *χ*<sup>p</sup> and the velocity-path relative angle *χ*<sup>r</sup> are defined as

$$\chi\_{\rm P} = \operatorname{atan2}(E\_{k+1} - E\_{k\prime}N\_{k+1} - N\_k) \,\tag{20}$$

$$\chi\_{\mathbf{r}}(e) = \arctan(-\mathcal{K}\_{\mathbf{P}}e - \mathcal{K}\_{\mathbf{I}} \int\_{0}^{t} e(\tau)d\tau),\tag{21}$$

where (*N<sup>k</sup>* , *E<sup>k</sup>* ) and (*Nk*+<sup>1</sup> , *Ek*+<sup>1</sup> ) are the positions of the passed and next waypoint, respectively, the proportional gain is *K*<sup>P</sup> = 1/∆(*t*) > 0, and *K*<sup>I</sup> > 0 represents the integral gain. The cross-track error *e*(*t*) is given by

$$e(t) = -[N\_{\rm AUV}(t) - N\_{\rm k}]\sin(\chi\_{\rm P}) + [E\_{\rm AUV}(t) - E\_{\rm k}]\cos(\chi\_{\rm P}).\tag{22}$$