**Synthetic Aperture Radar (SAR) Meets Deep Learning**

Editors

**Tianwen Zhang Tianjiao Zeng Xiaoling Zhang**

MDPI • Basel • Beijing • Wuhan • Barcelona • Belgrade • Manchester • Tokyo • Cluj • Tianjin

*Editors* Tianwen Zhang University of Electronic Science and Technology of China China

Tianjiao Zeng University of Hong Kong Hong Kong

Xiaoling Zhang University of Electronic Science and Technology of China China

*Editorial Office* MDPI St. Alban-Anlage 66 4052 Basel, Switzerland

This is a reprint of articles from the Special Issue published online in the open access journal *Remote Sensing* (ISSN 2072-4292) (available at: https://www.mdpi.com/journal/remotesensing/ special issues/synthetic aperture radar meets deep learning).

For citation purposes, cite each article independently as indicated on the article page online and as indicated below:

LastName, A.A.; LastName, B.B.; LastName, C.C. Article Title. *Journal Name* **Year**, *Volume Number*, Page Range.

**ISBN 978-3-0365-6382-4 (Hbk) ISBN 978-3-0365-6383-1 (PDF)**

© 2023 by the authors. Articles in this book are Open Access and distributed under the Creative Commons Attribution (CC BY) license, which allows users to download, copy and build upon published articles, as long as the author and publisher are properly credited, which ensures maximum dissemination and a wider impact of our publications.

The book as a whole is distributed by MDPI under the terms and conditions of the Creative Commons license CC BY-NC-ND.

## **Contents**

## **Preface to "Synthetic Aperture Radar (SAR) Meets Deep Learning"**

A synthetic aperture radar (SAR) is an important active microwave imaging sensor, whose all-day and all-weather working capacity give it an important place in the remote sensing community. Since the United States launched the first SAR satellite, SAR has received much attention in the remote sensing community, e.g., in geological exploration, topographic mapping, disaster forecast, and traffic monitoring. It is valuable and meaningful, therefore, to study SAR-based remote sensing applications.

In recent years, deep learning represented by convolution neural networks has promoted significant progress in the computer vision community, e.g., in face recognition, the driverless field and Internet of things (IoT). Deep learning can enable computational models with multiple processing layers to learn data representations with multiple-level abstractions. This can greatly improve the performance of various applications. Today, scholars are realizing the potential value of deep learning in remote sensing. Many remote sensing application techniques incorporate deep learning, e.g., in target and oil spill detection, traffic surveillance, topographic mapping, AI-based SAR imaging algorithm updating, coastline surveillance, and marine fisheries management.

Interestingly, when SAR meets deep learning, how to use this advanced technology correctly needs to be considered carefully, and how to achieve the best performance of this "black-box" model also needs careful consideration. Notably, deep learning uncritically abandons traditional hand-crafted features and relies excessively on the abstract features of deep networks. Is this reasonable? Can the abstract features of deep networks fully represent a real SAR? Should the traditional hand-crafted features provided with mature theories and elaborate techniques be abandoned completely? These questions are worth pondering when one applies various deep learning techniques to the SAR remote sensing community. In general, deep learning is always proposed for natural optical images whose imaging mechanisms are greatly different from those of SARs.

When SAR meets deep learning, should an SAR adapt itself to deep learning, or should deep learning adapt itself to an SAR? The relationship between the two needs further exploration and research. Furthermore, is deep learning really suitable for SARs? The number of SAR samples is far smaller than that of natural optical images. In this case, could we ensure that deep networks are able to thoroughly learn SAR mechanisms?

This Special Issue provides a platform for researchers to handle the above significant challenges and present their innovative and cutting-edge research results when applying deep learning to SARs in various manuscript types, e.g., articles, letters, reviews and technical reports.

> **Tianwen Zhang, Tianjiao Zeng, and Xiaoling Zhang** *Editors*

### *Editorial* **Synthetic Aperture Radar (SAR) Meets Deep Learning**

**Tianwen Zhang 1, Tianjiao Zeng 2,3 and Xiaoling Zhang 1,\***

#### **1. Introduction**

Synthetic aperture radar (SAR) is an important active microwave imaging sensor. Its all-day and all-weather working capacity makes it play an important role in the remote sensing community. Since the launch of the first SAR satellite by the United States [1], SAR has received extensive attention in the remote sensing community [2], e.g., geological exploration [3], topographic mapping [4], disaster forecast [5,6], and marine traffic management [7–10]. Therefore, it is valuable and meaningful to study SAR-based remote sensing applications [11].

In recent years, with the rapid development of artificial intelligence, deep learning (DL) [12] has been applied to all walks of life, such as face recognition, automatic driving, search recommendation, internet of things, and so on. The DL represented by convolutional neural network (CNN) is promoting the evolution of many algorithms and the innovation of advanced technologies. At present, scholars are exploring the application value of DL in SAR remote sensing field. Many SAR remote sensing application technologies based on DL have emerged, such as land surface change detection, ocean remote sensing, sea-land segmentation, traffic surveillance and topographic mapping.

Aiming to promote the application of DL in SAR, we initiated this Special Issue and collected a total of 14 papers (including 12 articles, 1 review and 1 technical note) covering various topics, e.g., object detection, classification and tracking, SAR image intelligent processing, data analytics in the SAR remote sensing community and interferometric SAR technology. The overview of contribution is in the following section.

#### **2. Overview of Contribution**

On the topic of object detection, classification and tracking, Li et al. [13] summarized the dataset, algorithm, performance, DL framework, country and timeline of DL-based ship detection methods. They analyzed the 177 published papers about DL-based SAR ship detection and attempted to stimulate more research in this field. Xia [14] proposed a visual transformer framework based on contextual joint-representation learning referred to as CRTransSar. CRTransSar combined the global contextual information perception of transformers and the local feature representation capabilities of convolutional neural networks (CNNs). It was found to produce more accurate ship detection results than other most advanced methods. Note that the authors also released a larger-scale SAR multiclass target detection dataset called SMCDD. Feng et al. [15] established a lightweight position-enhanced anchor-free SAR ship detection algorithm called LPEDet. They designed a lightweight multiscale backbone and a position-enhanced attention strategy for balancing detection speed and accuracy. The results showed that their method achieved a higher detection accuracy and a faster detection speed than other state-of-the-art (SOTA) detection methods. Xu et al. [16] presented a unified framework combining triangle distance IoU loss

**Citation:** Zhang, T.; Zeng, T.; Zhang, X. Synthetic Aperture Radar (SAR) Meets Deep Learning. *Remote Sens.* **2023**, *15*, 303. https://doi.org/ 10.3390/rs15020303

Received: 6 December 2022 Accepted: 3 January 2023 Published: 4 January 2023

**Copyright:** © 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

(TDIoU loss), an attention-weighted feature pyramid network (AW-FPN), and a Rotated-SARShip dataset (RSSD) for arbitrary-oriented SAR ship detection. Their method showed superior performance on both SAR and optical image datasets, significantly outperforming the SOTA methods. Xiao et al. [17] proposed a simple, yet effective, self-supervised representation learning (Lite-SRL) algorithm for the scene classification task. Note that they successfully evaluate the on-board operational capability of Lite-SRL by transplanting Lite-SRL to the low-power computing platform NVIDIA Jetson TX2. Kaˇcan et al. [18] explored object classification on a raw and a reconstructed Ground-based SAR (GBSAR) data. They revealed how processing raw data provides overall better classification accuracy than processing reconstructed data, and revealed the value of this method in industrial GBSAR applications where processing speed is critical. Bao et al. [19] proposed a guided anchor Siamese network (GASN) for arbitrary targets of interest (TOI) tracking in Video-SAR. GASN used a matching function for returning the most similar area, followed by a guided anchor subnetwork to suppress false alarms. GASN realized the TOI tracking with high diversity and arbitrariness, outperforming SOTA methods.

On the topic of SAR image intelligent processing, Tan et al. [20] proposed a featurepreserving heterogeneous remote sensing image transformation model. Through decoupling network design, the method enabled enhancing the detailed information of the generated optical images and reducing its spectral distortion. The results in SEN-2 satellite images revealed that the proposed model has obvious advantages in feature reconstruction and the economical volume of the parameters. Zhang et al. [21] proposed a self-supervised despeckling algorithm with an enhanced U-Net called SSEUNet. Unlike previous selfsupervised despeckling works, the noisy-noisy image pairs in SSEUNet were generated from real-world SAR images through a novel generation training pairs module, making it possible to train deep convolutional neural networks using real-world SAR images. Finally, experiments on simulated and real-world SAR images show that SSEUNet notably exceeds SOTA despeckling methods. Habibollahi et al. [22] proposed a DL-based change detection algorithm for bi-temporal polarimetric SAR (PolSAR) imagery called TCD-Net. In particular, this method applied three steps as follows: (1) pre-processing, (2) parallel pseudo-label training sample generation based on a pre-trained model and the fuzzy C-means (FCM) clustering algorithm, and (3) classification. TCD-Net could learn more strong and abstract representations for the spatial information of a certain pixel, and was superior to other well-known methods. Fan et al. [23] proposed a high-precision, rapid, large-size SAR image dense-matching method. The method mainly included four steps: down-sampling image pre-registration, sub-image acquisition, dense matching, and the transformation solution. The experimental results demonstrated that the proposed method is efficient and accurate, which provides a new idea for SAR image registration. Zhang et al. [24] proposed A low-grade road extraction network Based on the fusion of optical and SAR images at the decision level called SDG-DenseNet. Furthermore, they verified that the decision-level fusion of road binary maps from SAR and optical images can significantly improve the accuracy of low-grade road extraction from remote sensing images.

On the topic of data analytics in the SAR remote sensing community, Wangiyana et al. [25] explored the impact of several data augmentation (DA) methods on the performance of building detection on a limited dataset of SAR images. Their results showed that geometric transformations are more effective than pixel transformations and DA methods should be used in moderation to prevent unwanted transformations outside the possible object variations. The study could provide potential guidelines for future research in selecting DA methods for segmentation tasks in radar imagery.

On the topic of interferometric SAR technology, Pu et al. [26] proposed a robust least squares phase unwrapping method called PGENet that works via a phase gradient estimation network based on the encoder–decoder architecture for InSAR. Experiments on simulated and real InSAR data demonstrated that PGENet outperformed the other five well-established phase unwrapping methods and was robust to noise.

#### **3. Conclusions**

Recently, as many SAR systems have been put into use, massive SAR data are available, providing important support for exploring how to apply DL to SAR fields. A large number of SAR data coupled with the DL methodology jointly promote the development of SAR fields. The Special Issue shows innovative applications in object detection, classification and tracking, SAR image intelligent processing, data analytics in the SAR remote sensing community and interferometric SAR technology. There is no doubt that applying DL to more SAR fields (such as terrain classification, SAR agriculture monitoring, SAR imaging algorithm updating, SAR forest applications, marine pollution, etc.) is of great significance for earth remote sensing. In addition, we welcome scholars who are interested in applying DL to SAR to contribute to the scientific literature on this subject.

**Author Contributions:** Writing—original draft preparation, T.Z. (Tianwen Zhang); writing—review and editing, T.Z. (Tiaojiao Zeng) and X.Z. All authors have read and agreed to the published version of the manuscript.

**Data Availability Statement:** Not applicable.

**Acknowledgments:** We thank all authors, reviewers and editors for their contributions.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

**Disclaimer/Publisher's Note:** The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

## *Technical Note* **Data Augmentation for Building Footprint Segmentation in SAR Images: An Empirical Study**

**Sandhi Wangiyana \*, Piotr Samczy ´nski and Artur Gromek**

Institute of Electronic Systems, Faculty of Electronics and Information Technology, Warsaw University of Technology, 00-665 Warsaw, Poland<EMAIL_ADDRESS>(P.S<EMAIL_ADDRESS>(A.G.) **\*** Correspondence<EMAIL_ADDRESS>
**Abstract:** Building footprints provide essential information for mapping, disaster management, and other large-scale studies. Synthetic Aperture Radar (SAR) provides consistent data availability over optical images owing to its unique properties, which consequently makes it more challenging to interpret. Previous studies have demonstrated the success of automated methods using Convolutional Neural Networks to detect buildings in Very High Resolution (VHR) SAR images. However, the scarcity of such datasets that are available to the public can limit research progress in this field. We explored the impact of several data augmentation (DA) methods on the performance of building detection on a limited dataset of SAR images. Our results show that geometric transformations are more effective than pixel transformations. The former improves the detection of objects with different scale and rotation variations. The latter creates textural changes that help differentiate edges better, but amplifies non-object patterns, leading to increased false positive predictions. We experimented with applying DA at different stages and concluded that applying similar DA methods in training and inference showed the best performance compared with DA applied only during training. Some DA can alter key features of a building's representation in radar images. Among them are vertical flips and quarter circle rotations, which yielded the worst performance. DA methods should be used in moderation to prevent unwanted transformations outside the possible object variations. Error analysis, either through statistical methods or manual inspection, is recommended to understand the bias presented in the dataset, which is useful in selecting suitable DAs. The findings from this study can provide potential guidelines for future research in selecting DA methods for segmentation tasks in radar imagery.

**Keywords:** image augmentation; building extraction; SAR; semantic segmentation

#### **1. Introduction**

Buildings are the main structures in any urban area. A building's footprint is a polygon surrounding a building's area when viewed from the top. Maintaining this geographic information is vital for city planning, mapping, disaster preparedness, or other large-scale studies. Synthetic Aperture Radar (SAR) imagery provides an advantage over optical sensors by penetrating through clouds and capturing data day-night and in all-weather conditions. This consistently available remote sensing data has attracted researchers to study areas frequently covered by clouds, such as in disastrous situations. Temporal changes are better identified using methods such as Change Detection in largescale areas [1]. However, its unique properties are difficult for non-experts to analyze. This fact leads to the exploitation of automated methods such as deep learning using the Convolutional Neural Network (CNN).

CNN is known for extracting relevant features automatically by learning the underlying function that maps a pair of input and output examples. Automated building detection in Very High Resolution (VHR) SAR images was demonstrated using CNN in [2,3]. Despite this, detecting buildings in an urban SAR scenery is challenging due to the complex

**Citation:** Wangiyana, S.; Samczy ´nski, P.; Gromek, A. Data Augmentation for Building Footprint Segmentation in SAR Images: An Empirical Study. *Remote Sens.* **2022**, *14*, 2012. https:// doi.org/10.3390/rs14092012

Academic Editors: Bahram Salehi, Tianwen Zhang, Tianjiao Zeng and Xiaoling Zhang

Received: 2 March 2022 Accepted: 20 April 2022 Published: 22 April 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

background and multi-scale objects [2]. In an urban area, buildings are visible but difficult to distinguish from each other. The major indicator of a building is the double bounce scattering formed by the grounds and walls that are visible to the sensor. However, in large cities, high-rise buildings can be challenging to detect due to a phenomenon called layover, which projects the building's wall at the ground toward the sensor. A different approach was taken in [4], where a CNN model was trained to predict these layover areas instead of the traditional building footprint. Predicting the visible building region is indeed intuitive in SAR images, but for most GIS applications, the building footprints are still desirable. An extensive search space on various architectures, pre-trained weights, and loss functions for segmenting building footprints from optical and SAR images was performed in [5]. It was found that the diverse building areas and heights in different cities were problematic. Small-area buildings, mostly found in Shanghai, Beijing, and Rio, were undetectable, while high-rise buildings (mostly in San Diego and Hong Kong) degraded the model's performance due to extreme geometric distortions. Those models performed well in cities such as Barcelona and Berlin because most of the buildings were of moderate size and height.

Predicting well on unseen data or the ability to generalize is the main goal of training a deep learning model. It is a generally accepted fact that deep neural networks perform well on computer vision tasks by relying on large datasets to avoid overfitting [6]. Overfitting happens when a model fits its training set too well. This results in low accuracy predictions on novel data. For the task of building footprint extraction, a handful of datasets from optical sensors exist [7,8], but unfortunately, not many datasets with VHR SAR data are available for public usage. Open, high-quality datasets can be used as a standard benchmark to compare different algorithms and methods. The release of the SpaceNet6 dataset [9] was aimed to promote further research on this topic.

For data that are expensive to collect and label, such as radar or medical images, a common technique to boost performance is using data augmentation. Data augmentations (DA) increase the set of possible data points, artificially growing the dataset's size and diversity. It potentially helps the model avoid focusing on features that are too specific to the data used for training, therefore, increasing generalization (the ability to predict well on data not seen during training) without the need to acquire more images [10].

The use of data augmentation for CNN models has proved effective for classifying rare, deadly skin lesions through basic geometric transformations [11] and unique data cleaning methods [12]. In remote sensing, [13] investigated data augmentations for hyperspectral remote sensing images during inference, while [14] performed object augmentation to increase the number of buildings in optical remote sensing images and demonstrated better building extraction performance. In SAR imagery, [15] showed improvements in paddy rice semantic segmentation by applying quarter-circle rotations and random flipping. Random erasing [16] on target ships was performed in [17] to simulate information loss in radar imagery and improve the robustness of object detection. Conventional methods can be used as a form of DA, as demonstrated in [18], which combined handcrafted features from a Histogram of Oriented Gradients (HOG) with automated features from CNN to improve ship classification in SAR imagery.

Generative models are also a popular data augmentation method, by generating synthetic samples that still retain similar characteristics to the original set. It uses an architecture called Generative Adversarial Network (GAN). However, due to the high computation cost, these are generally applied for low-resolution images, such as in target recognition [19,20], or in a limited scene understanding, such as reducing speckle filters [21,22]. Another promising form of adding synthetic data for SAR is by using simulations. In [23], Inverse SAR (ISAR) images were generated using a simulation to augment the limited SAR data for marine floating raft aquaculture detection. Variations in imaging methods and environmental conditions can be consistently created while retaining accurate labels and information about the target. These would be expensive to both collect and analyze using direct measurements. However, the current state-of-the-art simulators are still

unable to produce high fidelity images that create a gap between measured and synthetic SAR imagery [24].

To the best of our knowledge, no prior research investigated and compared augmentation methods specifically for building detection from SAR images. In this study, we explored several geometric and pixel transformations and their performance on the SpaceNet6 dataset [9]. The article is organized as follows: Section 2 provides a technical overview of the dataset, CNN model, and training details. All data augmentation methods used in this research are briefly explained in Section 3. The results of the ablation study and the main experiments are presented in Section 4 and discussed in Section 5. Finally, we conclude the article in Section 6.

The main contributions of this paper are:

#### **2. Materials and Methods**

#### *2.1. Dataset Overview*

SpaceNet6 was released as a dataset for a competition to extract building footprints from multi-sensor data. The competition data consist of a training set and a testing set. The latter we do not use because there were no labels provided to verify our predictions. The former consists of 3401 tiles of satellite imagery, each from optical RGB, near-infrared (NIR), and SAR. Several works made use of this multi-modal data to boost segmentation performance [5,25,26]. In this study, we are only interested in using the SAR data. A quad polarization X-band sensor mounted on an aerial vehicle was used to take images over the largest port in Europe, the Rotterdam port. The 120 km2 coverage is split into tiles of 450 m × 450 m, with 0.5 m/pixel spatial resolution in both range and azimuth direction. An example of an image tile is shown in Figure 1.

**Figure 1.** A tile example shows a pair of (**a**) optical RGB and (**b**) SAR shown as false-colored (R = HH, G = VV, B = VH) over a small oil storage area. (**c**) shows the building footprints overlayed on top of the false-colored SAR. One main challenge of this dataset is differentiating patterns of buildings against objects with high backscatter intensity (such as containers, oil silos, and ships) in the port area.

The SAR data has 2 orientations; both were captured using a north and a south facing sensor. Figure 2a shows the tile over a base map of Rotterdam city, marking the position of images from orient1 (or north-facing) in green and orient0 (or south-facing) in red. The direction of flight is indicated by the azimuth (*az*) arrow, while the direction where the sensor is facing is given by the range (*rg*) arrow.

**Figure 2.** (**a**) The map of Rotterdam Port area (UTM Zone 31N) overlayed with tile boundaries for all 3401 tiles in the SpaceNet6 training set. Green tiles are orient1 while red tiles are orient0. To showcase the data augmentation methods in this study, we only use the green tiles (orient1) for training. Some building footprints are highlighted as an example of layover effects in (**b**) orient1 and (**c**) orient0 tiles. Notice the direction of a layover is always projected towards the sensor's position (near range) while the shadow is cast away from the sensor. The optical RGB image is provided for comparison.

Each orientation creates different characteristics of how a building looks, namely the shadows and layover. The layover creates a projection of the building's body onto the ground. It is caused by the side-looking nature of SAR imaging and the fact that SAR imaging maps pixel values based on the order of received echo. This is especially prominent in tall buildings.

The quad-channel SAR (also known as polarimetric SAR) is excellent for deep learning applications that benefit from the extra data. However, to showcase the effectiveness of data augmentation methods, we only used tiles from orient1 (covering the northern part of the city) and only a single channel HH polarization. This constraint enforces overfitting due to a limited data situation while still maintaining distinct land use cover of the port area and residential buildings. We chose only the HH polarization channel due to it being mostly available in SAR constellations such as GaoFen3, TerraSARX, and Sentinel 1 [27].

#### *2.2. Preparing the Dataset*

The purpose of training a Deep Learning model is to predict well on new data, which the model never sees during training. This is called generalization. The proper way to develop a model is to prepare 3 separate datasets called the training set, the validation set, and the testing set. As their names suggest, each is used in a different phase of a model's development. For each iteration of training, called an epoch, the model updates its weights by learning from the training set. At the end of each epoch, the model is tested on the validation set without updating its weights. This is used as a guideline for the Deep Learning practitioner to optimize the model or training parameters by studying the generalization ability over each epoch. Finally, when training is finished, the true performance of a model is determined by its score on a separate testing set.

To train our model, we used roughly half of the SpaceNet6 training set, which was constrained to a single sensor orientation and a single HH channel. We did not use a separate testing set. To evaluate the performance of our model and as a guideline for the impact of augmentation methods, we generated a validation dataset using the expanded version of SpaceNet6. This extra data was released later after the SpaceNet6 competition finished and consisted of unprocessed Single Look Complex (SLC) SAR data with additional polygon labels. We generated tile images from the SLC rasters with the same constraint as our training set: a single orientation and a single HH channel. We matched the exact preprocessing steps explained in [9] using their provided Python library. The steps are:

**Figure 3.** One of the SLC stripes after preprocessed and cropped to the boundaries of the testing set labels. These will be further cropped into non-overlapping tiles and later used as our validation dataset.

We used a search algorithm to crop and remove the no-data regions. These are the black regions shown in the left and bottom parts of the image tile in Figure 1. They were produced during the pre-processing stage of the dataset, stemming from the need to have a square non-overlapping tile image. The borders between the raster and the no-data regions created jagged lines due to slight affine rotation during orthorectification. Our experiments have shown these jagged lines to affect the results of pixel transformations, so in order to obtain consistent results, we have cropped them to a clean rectangle shape tile. The code we used in our methods is available as open source in https://github.com/sandhi-artha/sn6\_aug (last accessed on 9 April 2022).

#### *2.3. Segmentation Model*

Segmentation is the task of partitioning the image into regions based on the similarity (alike characteristic within the same class) and discontinuity (the border or edge between different classes) of pixel intensity. For building footprint extraction, there are only 2 classes: the positive examples, i.e., pixels belonging to a building's region, and negative examples, which are the rest of the pixels (non-building). The deep learning model is given a pair of images and labels for training. The iterative process outputs a similar-sized image classifying each pixel as belonging to one of the two classes.

An encoder-decoder type architecture is commonly used for segmentation models, popularized by the famous UNet [28] and its variations. For aerial imagery, DeepLab v3 [29], PSPNet [30], and Feature Pyramid Network (FPN) [31] are commonly used. Based on a previous study [32], we used the FPN architecture combined with the EfficientNet B4

backbone. EfficientNet is a family of CNN models generated using compound scaling to determine an optimal network size [33]. For a deep learning model, the architecture refers to how each layer in the network is connected, while the backbone refers to the feature extraction part of the model.

Building footprints taken from overhead images have various sizes. To differentiate a building from the background, we need to see enough pixels representing the whole or most of the building. This means a higher spatial resolution is required for detecting buildings with a smaller area. A common method in computer vision to help the model learn these multi-sized objects is to use multi-scale input, i.e., the input image downscaled to different pixel resolutions. This is called an image pyramid (Figure 4).

**Figure 4.** FPN architecture with EfficientNet backbone for segmentation tasks.

FPN uses feature pyramids instead. In the Encoder, the image input is scaled down using a dilated convolution operation which cuts the image dimension in half at each pyramid level. As the data flows up the pyramid, the top layer will have the least width and height (the original input's size divided by 32) but the richest semantic information (1632 feature maps or channels). In a classification task, this is compressed further to output a vector with the same size as the number of classification labels [31]. For a segmentation task, an output with the same spatial size as the input is required. Therefore, the top layer needs to be upscaled.

A 1 × 1 convolution filter is applied to the final layer in the encoder pyramid to reduce the number of feature maps to 256, without modifying the image dimension. As data flows down the Decoder's pyramid, the width and height increase 2× using nearest neighbors upsampling. In the skip connections (yellow arrow), feature maps from the same pyramid level in the Encoder and Decoder were concatenated. A 1 × 1 convolution was used to scale the feature maps from the Encoder pyramid to 256. This provides context for better localization as the image gradually recovers in pixel resolution. Afterward, feature pyramids from the Decoder go through a Conv and Upsample operation (black arrow), resulting in modules with 128 feature maps and image dimension 1/4 of the original input. These are then stacked channel-wise, creating a module of 512 feature maps. A final Conv and Upsample operation reduces the number of channels to 1 and restores the image dimension back to the original input [34].

#### *2.4. Training Details*

The training was performed in a Kaggle Kernel, a cloud computing environment equipped with a 2-core processor and an Nvidia P100 GPU with 16 GB of video memory (VRAM). The training pipeline was built using the TensorFlow framework. The

Segmentation-Models library [35] was used to combine the FPN architecture with the EfficientNet B4 backbone with no pre-trained weights. Adam [36] was used as the optimizer with default parameters. We used a learning rate scheduler, which configures the learning rate *α* to gradually decay by a factor of 0.5 ∗ (1 + cos(*nπ*/*N*)) where *n* is the current epoch and *N* is the total number of epochs.

To evaluate the performance of our model, we use the Intersection over Union (IoU) as the metric. IoU is the ratio of overlapping between the predicted area and the real area (Figure 5). In this case, it is a pixel-based metric. A higher IoU indicates a better predictive accuracy.

$$\text{IoU} = \frac{\mathcal{Y}\_{\mathcal{S}^t} \cap \mathcal{Y}\_{pred}}{\mathcal{Y}\_{\mathcal{S}^t} \cup \mathcal{Y}\_{pred}} = \frac{\text{TP}}{\text{TP} + \text{FP} + \text{FN}} \tag{1}$$

**Figure 5.** How IoU is calculated over an optical image of a warehouse building.

True Positives (TP) are pixels labeled as building and are correctly predicted as building. True Negatives (TN) are pixels labeled as background and are correctly predicted. False Negatives (FN) are misclassified background pixels, while False Positives (FP) are misclassified pixels of buildings.

Calculating statistics over each image tile in the training set, 20% of tiles have less than 1% positive samples (pixels classified as buildings) (Figure 6a). This indicates that most tiles contain high negative samples (background pixels). One must be cautious in selecting a loss function for training a model on a skewed data distribution such as this because the negative samples will dominate the predictions. For example, using a binary cross-entropy as the loss function, the model will obtain a minor error even if it predicted the whole image as background pixels.

We experimented with several loss functions and concluded that Dice Loss leads to better convergence in this dataset. It is based on the Dice Coefficient, which is used to calculate the similarity between 2 samples based on the degree of overlapping. Dice loss is simply 1 − *Dice Coef ficient*. This results in a loss or error score ranging from 0 to 1, where 0 indicates a perfect and complete overlap.

$$Loss\_{Dict} = 1 - 2 \* \frac{y\_{\mathcal{S}^t} \cap \ y\_{pred}}{y\_{\mathcal{S}^t} + y\_{pred}} \tag{2}$$

**Figure 6.** Per image tile statistics for the training set, normalized. (**a**) shows the distribution of positive samples compared to the total number of pixels in an image tile and 20% of tiles have less than 1% of total pixels categorized as buildings. (**b**) shows the number of buildings count for each image tile. Most tiles (17.5%) contain less than 5 buildings.

#### *2.5. Ablation Study*

First, we studied the impact of each augmentation method in an ablation study, applying the same model and training configuration but applying different transformations to the dataset. To speed up this process, we trained and validated only a subset of our main training set. We divided it into 5 identical columned regions. The first and second columns were used as the mini-training dataset, while the last column was for the mini-validation dataset. We did not include the middle area in the mini training set to create a buffer (notice the high overlapping of tiles in Figure 2a) which prevents data leakage between both sets. The mini-training set and the mini-validation set contain 37% and 23% images of the main training set, respectively. After concluding which augmentation works well for the mini dataset in the ablation study, we applied combinations of positively impactful transformations to the main dataset.

#### **3. Data Augmentation**

This section describes the data augmentations used in this study and how they were implemented during the model's development. In general, the geometric transformations (including reduce transformation) were applied using TensorFlow operations, while pixel transformations were applied using the help of the Albumentation library [37].

#### *3.1. Types of Data Augmentation*

#### 3.1.1. Reduce Transformation

Cropping the no-data regions result in a varying aspect ratio for each tile that must be addressed. Square-shaped images are preferred as data input to simplify spatial compression and expansion of the image inside the model. The image size also needs to be reduced to fit into the GPU's memory. We decided on the target resolution of 320 pixels by 320 pixels, which allowed the batch size of eight for the single P100 GPU. All resizing methods used the bilinear interpolation. Two main resize methods were tested:

This downsampling process can be exploited to introduce randomness that further increases the diversity of the training samples. Cropping at random locations gave better details than just resizing the whole image (Figure 7). However, because it introduces randomness, these methods cannot be used as a reduction method for the validation dataset:

**Figure 7.** Reduce Transformations comparison.

#### 3.1.2. Geometric Transformations

In computer vision tasks, geometric transformations are cheap and easy to implement. However, it is important to be aware of choosing the transformations' magnitude that preserves the label in the image. For example, in optical character recognition, rotating a number by 180◦ can result in a different label interpretation in the case of the numbers six and nine.

Flipping an image along the horizontal or vertical centerline is a common data augmentation method. Referring to Figure 2a, the range direction *rg* for this dataset is on the vertical or *y*-axis, while the flight direction *az* is on the horizontal or *x*-axis. The **Horizontal Flip** does not alter the properties of a radar image. It would be as if the vehicle carrying the sensor was moving in the opposite direction. In contrast, the **Vertical Flip** makes the shadows and layovers appear on the opposite side, creating inconsistency.

Rotation helps the model learn the invariant orientation of a building. We performed **Rotation90** or quarter circle rotations {90◦, 180◦, 270◦} and **Fine Rotation** with a randomized angle range, e.g., [−10◦, 10◦]. Similar to Vertical Flip, the quarter circle rotation affects the imaging properties of the radar. The fine rotation exposes an area where image data are unknown. There are several ways to "fill" this blank area, and our experiments showed that leaving it to a dark pixel (0.0) gave the best result.

Shear is a distortion along a specific axis used to modify or correct perception angles (Figure 8). Despite SAR being a side-looking imaging device, the processed SAR image appears flat owing to the orthorectification process that corrects geometric distortions. In **ShearX**, the edges of the image that are parallel to the *x*-axis stay the same, while the other two edges are displaced depending on the shear angle range. **ShearY** is the exact opposite. We set the shear rotation to be randomized between an angle range of [−10◦,10◦].

**Figure 8.** Shear transformation in 2 directions.

**Random Erasing** is an augmentation method inspired by the drop-out regularization technique [16]. It simply selects a random patch or region in the image and erases the pixels within that region. The goal is to increase robustness to occlusion by forcing the model to learn an alternative way of recognizing the covered objects. We have decided to fill the patch values with a dark pixel (0.0). The random erasing was implemented using CoarseDropout class from the Albumentation library. The region's width and size were randomized from 30 pixels to 40 pixels, and the number of regions created was randomized from two to ten patches. The proposed geometric transformations are shown in Figure 9.

**Figure 9.** Summary of Geometric Transformations.

#### 3.1.3. Pixel Transformations

In airborne sensors, unknown perturbations of the sensor's position relative to its expected trajectory can cause several defects such as radiometric distortions and image defocusing. To increase the model's robustness when encountering these defects, we used noise injection methods. The common **Gaussian Noise** was generated using the GaussNoise class, while **Speckle Noise** was amplified by multiplying each pixel with random values using the MultiplicativeNoise class, both from the Albumentation library. Some images suffered from defocusing due to an unpredicted change in the flight trajectory, causing fluctuations in the microwave path length between the sensor and the scene [38]. We simulate this defocusing by applying **Motion Blur** with random kernel size using the MotionBlur class from Albumentation.

We used **Sharpening** with a high pass filter to improve edge detection. Consequently, this also increases other high-frequency components such as speckle.

The SAR images in this dataset are SAR-Intensity in the log scale, also referred to as dB images. From samples of the dataset, we observed that the dB images have a pixel distribution close to Rayleigh and mostly populate a narrow area. Figure 10 shows how histogram equalization methods stretch the pixel distribution to span the full range, increasing contrast and improving edge visibility. Contrast Limited Adaptive Histogram Equalization (**CLAHE**) works best when there are large intensity variations in different parts of the image, limiting the contrast amplification to preserve details. We applied it using the CLAHE class from the Albumentation library. The proposed pixel transformations are shown in Figure 11.

**Figure 10.** Histogram of a SAR Log Intensity image, before and after applying histogram equalization using CLAHE.

**Figure 11.** Summary of Pixel Transformations.

#### 3.1.4. Speckle Filters

Although still categorized as Pixel Transformation, this subsubsection was created to provide more explanation for Speckle Filters. Similar to other coherent imaging methods (laser, ultrasound), SAR suffers from noise-like phenomena called speckle. This happens due to the interference of multiple scatterers within a resolution cell. Speckle takes form in granular variations of pixel values that can lower the interpretability of a SAR image for computer vision systems as well as human practitioners.

Speckle reduction filters such as a Box filter can smooth the speckle using a local averaging window. This is effective in homogenous areas, but in applications requiring high-frequency information such as edges, filters that can adapt to local texture can better preserve information in heterogeneous areas [39]. A previous study [32] has shown a slight performance gain by applying Low Pass filters with varying strength on the UNet model. In this research, we inspected the use of well-known adaptive speckle filters as a form of data augmentation, namely Enhanced Lee Filter (**eLee**), **Frost** Filter, and Gamma Maximum A Posteriori (**GMAP**) Filter.

In Figure 12a, two sample crops of size 150 × 150 pixels are shown for comparing filtration results. A good filter should retain the average mean of an image while reducing speckle [40]. In homogenous areas, the standard deviation should ideally be 0. Speckle filters were applied in MATLAB to the SAR Intensity image (linear scale) and later converted back to the Log-Intensity image (dB scale). Results of filtering are shown in Figure 12c,d, while the quantitative comparison is presented in Table 1. We can observe that the GMAP filter was slightly better at preserving the average value and reducing variance in crops 1 and 2, respectively. The image's Equivalent Number of Looks (ENL) also showed a reduction in speckle variance for all filters.

**Figure 12.** A sample SAR Log Intensity (dB) Image (**a**) with its equivalent optical RGB (**b**). We further analyze two distinct crop regions of a building (crop 1) and a homogenous water area (crop 2). The results of filtering with selected speckle filters and various kernel sizes are given in (**c**,**d**). The image in linear scale appeared dark because of the wide dynamic range.

**Table 1.** Quantitative comparison of the applied filters. The mean, standard deviation (std), and Equivalent Number of Looks (ENL) are calculated in the linear scale.

#### *3.2. Data Augmentation Design and Strategy*

We set a probability of 50% for an image to load either the original version or with data augmentations applied. The magnitude of the transformation was also randomized in a value range, increasing the variation in every iteration of training, except for flipping and quarter circle rotation, which had a limited set of transformations.

The order of transformations is important when multiple augmentations are combined during the main experiment. Pixel transformations are applied first to prevent the presence of no-data pixels from affecting the results. Following it is a reduced transformation, and finally, a geometrical transformation. When using multiple pixel transformations, it is important to combine them into a "One Of" group. By chance, only one of the transformations will be applied, preventing the creation of a disastrous result. In geometric transformation, there is no grouping, so an image has a chance to go through all transformations, which might increase no-data pixels but are generally less harmful than multiple filtering operations.

As categorized in [10], there are three stages of applying Data Augmentation (DA): Online (on-the-fly), Offline, and Test Time Augmentation (TTA).

In Online DA, the input data is manipulated during training. This can lead to a bottleneck if a fast accelerator is used in training, but the augmentation algorithm is slow, leaving the accelerator mostly waiting for data. The advantage is that it does not store the inflated data in storage. On the other hand, Offline DA allows complex manipulation and will not bloat training time. However, since it is applied before training, it takes up storage, and the variations are pre-determined (less randomness). We used Offline DA only for speckle filtered images since they were processed outside the TensorFlow environment, and an image was stored for every applied filter. Other transformations in this study used Online DA, which can have a finer degree of randomness in every iteration.

In TTA, *A* additional images are generated from each test image *x*, where *A* is the number of augmentations applied during the inference or prediction stage. The model will then predict on *A* + 1 samples and the average sum will be taken as the final prediction. This method of predicting multiple transformed versions of the input mimics the theory of ensemble learning, where a group of models using different architectures or trained on different data combines their prediction to increase generalization. This was investigated in [41], concluding that TTA helped reduce overconfident incorrect predictions compared to when using only a single model.

In a classification task, averaging predictions are straightforward since the output is an array with a size equal to the number of classes. In a segmentation task, one must be cautious to perform augmentations that modify the location of labels (in our dataset, the building footprints). If such methods are used, the solution is simply to revert back to the transformation before averaging the predictions.

#### **4. Results**

#### *4.1. Ablation Study*

In this isolated experiment, we measured the impact of each augmentation method. The model was trained on the mini-training dataset and evaluated on the mini-validation dataset. Results are shown in Table 2. Loss and IoU are the scores for the training set, while Val Loss and Val IoU are scores from the validation dataset. The training lasted for 60 epochs. The four metrics were taken at the best epoch, which is when the model obtained the highest Val IoU. We chose not to take the final score at epoch 60 to showcase the best performance of each augmentation objectively. The score at the last epoch was always worse than the best epoch because the training scores kept getting better while the validation scores stagnated or even deteriorated. This can be visualized in Figure 13. The gap between both training and validation scores was less when an augmentation method was applied, which delayed overfitting, enabling the model to move into what is known as a Local Minima or a temporary performance peak.

**Table 2.** Results of Ablation study. All scores are in percentage units. For loss, the lower is better. For IoU, the higher the better. Scores are color-coded in comparison to Pad Resize, where green color projects positive gain while red projects negative gain.

To fit the image resolution to the model's input, we used two basic Reduce methods: Pad Resize and Distorted Resize. A slightly better score obtained using Distorted Resize might indicate that the black paddings were distracting the model because the full scope of the raster's space was not utilized. However, a problem arises when reshaping back the distorted prediction to the original aspect ratio, so we used Pad Resize as the Reduce method for the validation data. Adding randomness to the Reduce method using Random Crop Resize had the biggest performance gain compared to other augmentations. Randomizing the crop size gives the chance to see the image at different scales and details. Random Crop

did not perform well due to the small static crop size of a 160 m × 160 m area, increasing the chance of encountering partial parts of a building.

**Figure 13.** IoU scores comparing the four Reduce Transformations. The solid lines show training IoU scores, while the dashed lines show validation IoU scores. The loosely dotted vertical lines show where the best epoch (highest Val IoU) for a given method. Adding more variations to the input delays the overfitting and is shown by a later best epoch.

Geometric transforms generally increased performance, except for Vertical Flip and Rotation90. Both were detrimental to the performance as they caused the extreme displacement of shadows and layovers' location compared to the actual building footprint. Pixel transforms were not as effective, giving similar or slightly worse scores than the baseline. These augmentations affect the recognition of texture, an important feature when the edges of a building or its shape are unrecognizable due to occlusion or noise. However, this filtering can also be destructive as it also amplifies non-building patterns.

Training scores were generally lower when augmentations were applied, as the model struggles to find the underlying function among the additional variations. A strong increase in training scores for the GMAP speckle filter indicates better recognition of the training data. However, these variations were not shown among the validation data, hence the lower validation scores.

To validate the effects of our proposed data augmentation, we performed the Ablation Study on different mini datasets which we briefly described in the following:

Figure 14 shows that performance gain and loss on the other datasets mostly agree with results in SAR orient1. Random Crop Resize, Horizontal Flip, and Fine rotation showed consistent gains over all datasets. Meanwhile, Rotation90 showed consistent dips in performance, which are more prominent in datasets from SpaceNet6. The result from PAN highlights the method's impact on a similar geographic region (Rotterdam Port) but a different modality, while the result from Inria highlights the impact when exposed to the different urban settlements of multiple cities. However, due to the stochastic nature of deep neural networks, using an optimized model and training method fitted to one dataset might not translate to an optimal solution on another dataset, which has

different distribution [6]. Therefore, these directive insights should be further tweaked when working on a different dataset.

#### *4.2. Main Experiment*

Using similar concepts to those from the Ablation study, we experimented with several combinations of augmentation methods applied to the main training set and evaluated on our prepared validation set. The same model and training parameters were used except for the increased training duration to 90 epochs. This compensates for the additional variation and gives more room for the model to learn. A callback was set to monitor the best Val IoU score and labeled it as the best epoch. The following augmentation schemes were applied:

Only Baseline used Pad Resize as the Reduce method, while the other combinations used Random Crop Resize. For every augmentation scheme, the model's performance was taken at the best epoch and shown in Table 3**.** Due to different datasets, the scores in Table 2 should not be directly compared to results from this main experiment. The training time *ttrain* was measured as the duration of training but shown in the table as a scaling factor compared to Baseline where *ttrain* <sup>=</sup> *ttrain*\_*scheme ttrain*\_*baseline* .

In line with results from the Ablation Study, geometric transformations had better scores than pixel transformations. Increasing the magnitude of transformation did not lead to an increase in performance, as shown by the lower scores obtained in Heavy Geometry. Increasing the diversity of transformations in Combination also did not improve performance despite consisting of transformations that showed positive impacts during the Ablation Study.

All models predicted well on medium height elongated residential buildings (Figure 15a). Applying augmentation increases confidence, modeling a more accurate shape characterized by rooftop patterns. However, fine details of a building structure and small buildings remained undetected.

**Table 3.** Results of combining multiple augmentations. All loss and IoU scores are in percentage. *ttrain* is the scale of training time compared to the Baseline. A lower value indicates faster training time. For loss, lower is better. For IoU, higher is better. Scores are color-coded where a darker green indicates a better value.

**Figure 15.** Comparison of predictions from the trained models of different scene objects: (**a**) mediumheight residential buildings, (**b**) containers in a shipping port, (**c**) outdoor sports field, and (**d**) highrise buildings.

For an image tile of large negative samples (pixels belonging to non-building), pixel augmentations drive extra attention to high backscattering objects such as container storage and large shipping/port equipment made of metal (Figure 15b). This leads to an increase in false positives. Geometric augmentations were less prone to this. However, Geometric augmentations overfit non-building objects with building-shaped backscatters, such as the fences surrounding a sports field (Figure 15c). A combination of geometric and pixel augmentations seems to tune down these false positives and correctly recognize them as non-object patterns.

Occlusion was the biggest problem, especially related to high-rise buildings in dense areas (Figure 15d). All models failed to recognize buildings occluded by the overlay of a neighboring high-rise building. Interestingly Geometric augmentations tend to classify the overlayed parts as positives.

Despite not using any augmentations in Baseline, the Light Pixel scheme, applied using the Albumentation library, managed to train slightly faster. This was caused by the callback function, saving the state of the model each time it sees a better-monitored value, requiring additional time on certain epochs. It shows that training duration is not an objective measurement due to the randomness involved. However, we still included it as a comparison. The other schemes show that adding more transformation methods will increase training time.

#### *4.3. Test-Time Augmentation*

The state of the models in the Main Experiment was saved at their best epoch, and TTA was applied after the training ended. We experimented with two methods for TTA:

We applied TTA to the Baseline model and the best model from the Main Experiment, which was trained on the Light Geometry scheme. TTA comes at the cost of additional inference time *ttest*, which is a scaling factor compared to the Baseline's inference time. It increases proportionally to the number of augmentations applied. Compared to the time required by re-training a model, the additional inference time to implement TTA was negligible. Results for TTA are shown in Table 4.

**Table 4.** Results of applying TTA to the Baseline model and Light Geometry. All loss and IoU scores are in percentage. For loss, lower is better. For IoU, higher is better. Scores are color-coded where a darker green indicates a better value.

The Baseline and Light Geometry model benefit from TTA\_1, which consist of simple Geometric transformations. Interestingly, when TTA\_2 was applied to the Baseline model, the performance was lower, as it predicted fewer positive samples from the two square patches. The Baseline model, trained on images with a fixed scale, had less confidence in

predicting medium-sized buildings compared to the Light Geometry model, which had the chance to see variations of scaling thanks to the Random Crop Resize reduction method.

#### **5. Discussion**

Deep learning methods use vast parameter exploration. Results vary with different data, models, and training parameters (also known as hyperparameters). Our goal is to provide intuition based on our experiments of modifying the data while maintaining the model and hyperparameters unchanged. Radar images, including SAR, have unique properties that differ from optical images. Therefore, several transformations can result in poor performance. Selecting augmentation methods requires knowledge of the biases in the training data, either through statistical analysis, in the case of a large dataset, or through the manual inspection of samples. This helps reduce the search space instead of trying every available method.

Tiling is required in remote sensing images as it is impossible to fit a large raster directly to a model. The choice of target resolution will affect the detection of multi-scale objects, such as buildings. Introducing randomness by varying the scale and crop size during dataset loading is a cheap way of boosting performance since there is no need to store extra images, such as in the case of tiling with overlapping regions. However, cropping too much will increase the chance of a large object covering the whole space and hinder performance. No-data regions are inevitable when tiling a large raster, and in our experience, it is better to remove them before feeding the image tile to the model.

Our study shows that pixel transformations are not as effective as geometric transformations. The reason might be that kernel filters, which are the base of most pixel transformations, are already an integral component of the CNN model itself, thus, learnable by an adequately sized model.

Applying transformations at different stages of the model development has different tradeoffs. For instance, Offline DA can utilize the training images with their stored variations, all at once to train the model. We experimented on a dataset transformed with speckle filters and achieved similar scores to Random Crop Resize in Table 2 but at the cost of 3.5× training time. This might be a good option when the data consist of only a few hundred samples and gathering extra data is unfeasible, such as in the case of classifying rare medical images.

We demonstrated that Test Time Augmentations (TTA) is a cheap method to boost test scores. However, applying a set of augmentations during the test did not achieve better scores when compared to applying the same set of augmentations during training. The model predicted the varying test samples better, had it seen these variations during training. Therefore, applying augmentations in both stages will result in better scores. When using Shear and Fine Rotation in TTA, the angle must be kept low because it removes some portion of the image (outside the image boundary) where it will not return when doing the inverse transformation after prediction. This is why quarter-circle rotations and flips are more commonly used as TTA because they retain the full image after inverse transformation.

#### **6. Conclusions**

This paper presents several data augmentation methods for semantic segmentation of building footprints in SAR imagery. By artificially increasing the training dataset, we improved the model's generalization on unseen samples from the validation set, thereby reducing overfitting. The results show a 5% increase in Val IoU score when comparing the best augmentation scheme to the baseline model (no augmentation). Data augmentation can be very helpful in situations with limited data, either due to proprietary licenses or an expensive collection process.

For building detection in SAR, geometric transformations were more effective than pixel transformations. However, some transformations (such as vertical flip and quarter circle rotations) that alter key features of a building in SAR images were proven to be detrimental. Therefore, data augmentations must not be overused, especially since it takes more resources to train (either storage or processing time), which does not always lead to a better result. Test Time Augmentations showed an additional performance gain compared to augmentations applied only during training.

We hope this study can be used as a guide for future research to optimize object detection in a limited set of radar imagery or as an inspiration for investigating alternative methods to augment radar images. The search for effective data augmentation methods can be expensive; thus, automated approaches can save time and computing resources. These approaches have not yet been studied in this article. Furthermore, generating new data through generative models or the use of simulations remain interesting avenues to explore.

**Author Contributions:** Conceptualization, S.W. and P.S.; methodology, S.W.; software, S.W. and A.G.; validation, S.W., P.S. and A.G.; formal analysis, S.W.; investigation, S.W.; resources, S.W.; data curation, S.W.; writing—original draft preparation, S.W.; writing—review and editing, S.W., P.S. and A.G.; visualization, S.W.; supervision, P.S. and A.G.; project administration, S.W. and P.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

## *Review* **Deep Learning for SAR Ship Detection: Past, Present and Future**

**Jianwei Li 1, Congan Xu 1,2,\*, Hang Su 1, Long Gao <sup>1</sup> and Taoyang Wang <sup>3</sup>**

	-<EMAIL_ADDRESS>
**\*** Correspondence<EMAIL_ADDRESS>
**Abstract:** After the revival of deep learning in computer vision in 2012, SAR ship detection comes into the deep learning era too. The deep learning-based computer vision algorithms can work in an end-to-end pipeline, without the need of designing features manually, and they have amazing performance. As a result, it is also used to detect ships in SAR images. The beginning of this direction is the paper we published in 2017BIGSARDATA, in which the first dataset SSDD was used and shared with peers. Since then, lots of researchers focus their attention on this field. In this paper, we analyze the past, present, and future of the deep learning-based ship detection algorithms in SAR images. In the past section, we analyze the difference between traditional CFAR (constant false alarm rate) based and deep learning-based detectors through theory and experiment. The traditional method is unsupervised while the deep learning is strongly supervised, and their performance varies several times. In the present part, we analyze the 177 published papers about SAR ship detection. We highlight the dataset, algorithm, performance, deep learning framework, country, timeline, etc. After that, we introduce the use of single-stage, two-stage, anchor-free, train from scratch, oriented bounding box, multi-scale, and real-time detectors in detail in the 177 papers. The advantages and disadvantages of speed and accuracy are also analyzed. In the future part, we list the problem and direction of this field. We can find that, in the past five years, the AP50 has boosted from 78.8% in 2017 to 97.8 % in 2022 on SSDD. Additionally, we think that researchers should design algorithms according to the specific characteristics of SAR images. What we should do next is to bridge the gap between SAR ship detection and computer vision by merging the small datasets into a large one and formulating corresponding standards and benchmarks. We expect that this survey of 177 papers can make people better understand these algorithms and stimulate more research in this field.

**Keywords:** SAR ship detection; SAR dataset; single-stage detector; two-stage detector; anchor free; train from scratch; oriented bounding box; multi-scale detection; deep learning; computer vision

#### **1. Introduction**

Synthetic aperture radar (SAR) remote sensing has become one of the important methods for marine monitoring due to its all-day, all-weather advantage. Ship detection in SAR images has broad prospects in both military and civilian fields [1,2].

The traditional detection method includes three steps: sea-land segmentation, CFAR (constant false alarm rate) detection, and discrimination [3,4]. In the sea-land segmentation step, the land pixels are rejected to avoid interference with the CFAR step. The common method is based on GIS (geographic information system) or image features. The gray histogram is the classical feature used for segmentation. In the second step, CFAR is usually used for ship detection. The distribution function is assumed to fit the pixel distribution of the SAR image. K, Weibull, and Rayleigh distribution are usually used in this step. To keep the probability of a false alarm at a constant value, the CFAR algorithm compares

**Citation:** Li, J.; Xu, C.; Su, H.; Gao, L.; Wang, T. Deep Learning for SAR Ship Detection: Past, Present and Future. *Remote Sens.* **2022**, *14*, 2712. https://doi.org/10.3390/rs14112712

Academic Editor: Dusan Gleich

Received: 6 May 2022 Accepted: 2 June 2022 Published: 5 June 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

the testing pixel with an adaptive threshold that is generated by the local background surrounding the testing pixel. After the pre-screening by CFAR, a discriminator is needed to reject the background. Discriminator includes two procedures: feature designing and classifier designing. According to the feature difference between ship chips and non-ship chips, this step can reduce the number of false alarms. The traditional detection method dominated this field for a long time.

With the development of deep learning-based object detection algorithms in computer vision (CV) [5], SAR researchers also began to seek inspiration from computer vision. There are three reasons that can explain the revival of deep learning. They are the arising of computing power, big data, and corresponding algorithms. As SAR images are not easily accessed, the deep learning-based detection method cannot be used for SAR ship detection at the beginning.

This problem was solved in 2017, as the first dataset SSDD (SAR Ship Detection Dataset) was open to the public. SSDD provides the same data and evaluation criteria for researchers, and it solves the problem that the traditional algorithms lack data and are not comparable in this field. Since then, more and more researchers adopt a deep learningbased method in this area. The deep learning-based algorithms also show great results compared with the traditional CFAR-based method. The active and open characteristics of computer vision also further promote the development of this field. We think that the emergence of SSDD means that this field comes into the deep learning era.

As far as we know, there are 177 papers [6–182] that use deep learning-based algorithms to detect ships in SAR images. However, there are no papers that review them yet. In order to summarize the achievements of the 177 papers and show the way for the future, we specially wrote this paper, hoping to contribute to the development of this field.

The rest of this review is arranged as shown in Figure 1. Section 2 briefly analyzes some work related to our paper. Section 3 summarizes the past of the traditional detection algorithms in SAR images. It mainly includes CFAR, hand-crafted features, and limited shallow representation. Section 4 introduces the present in deep learning-based detectors. We review the 177 papers, divide them into 10 categories and analyze them, respectively. Section 5 shows the future direction of this field. Section 6 is the conclusion of the paper.

**Figure 1.** The overall architecture of the paper.

#### **2. Related Work**

As far as we know, few researchers have written review papers about this direction. This is partly due to the fact that this direction is new to some extent. At present, only three papers [105,127,170] have performed work related to our work.

Jerzy et al. [105] reviewed the papers from the last 5 years that discuss SAR ship detection. They mainly introduce the development of CFAR methods, CNN (convolutional neural network) based methods, GLRT (generalized likelihood ratio test) based methods, feature extraction-based methods, weighted information entropy-based methods, and variational Bayesian inference-based methods. Compared with paper [105], we mainly focus on the deep learning-based detection methods and do not focus on the traditional methods.

Mao et al. [127] solved the problem of the lack of performance benchmark for stateof-the-art methods on SSDD. Through this work, researchers can compare their work in the same experimental setup. They present 21 advanced detection models, including single-stage, two-stage, train from scratch detection algorithms, and so on. Compared with paper [127], we not only introduce the performance of different public datasets, but also classify all the papers, and summarize the principles and results of the algorithms.

Zhang et al. [170] solved the problem of the coarse annotations and ambiguous standards in SSDD. These improvements are beneficial for a fair comparison. It has played a great role in promoting the healthy development of this field. We suggest that researchers use the standards specified in this paper in the future. Compared with paper [170], our work is not limited to SSDD but introduces other datasets in this field. More importantly, our team has systematically analyzed, classified, and commented on the methods used, and pointed out the future research direction, which is beneficial to the development of this field.

In short, our work is different from the other papers. It is the first comprehensive review of SAR ship detection.

#### **3. Past—The Traditional SAR Ship Detection Algorithms**

Traditional detection algorithms in SAR images are based on hand-crafted features and limited shallow-learning representation. It can be divided into three steps: preprocessing, candidate region extraction, and discrimination.

CFAR is a common method for candidate region extraction. It can select potential ship regions. It first statistically models the clutter and then obtains the threshold value according to the false alarm rate. The pixels above the threshold are regarded as ship pixels, and those below the threshold are regarded as background. CFAR is essentially a segmentation-based algorithm, that is, the pixels are classified into two categories (ship or non-ship) according to the gray size, and then the ship pixel region is merged into the ship region. The performance of this method largely depends on the statistical modeling of sea clutter and the parameter estimation of the selected model. According to different SAR image products and practical application requirements, different statistical models such as Gaussian distribution, gamma distribution, log-normal distribution, Weibull distribution, and K distribution are proposed. Gaussian distribution and K distribution are the most commonly used. Generally speaking, when the scene is relatively simple, the CFAR method can achieve better results. However, for small ships and complex offshore scenes, due to the difficulty of modeling, it will have more false positives and poor detection performance.

Discrimination is generally realized by using artificially designed features and training classifiers. In addition to the simple features such as length, width, aspect ratio, and scattering point position, the features introduced from computer vision are also commonly used and have stronger robustness. Such as integral image features, HoG (histogram of oriented gradients), SURF (speeded up robust features), and LBP (local binary pattern). These features improve the performance of the detection algorithm. In classifier designing, decision trees, SVM, gradient boosting, and their improved versions also further improve the performance.

Feature and classifier designing have pushed this field forward in the past few years. However, since the rise of deep learning in 2012 [5], the above ideas are dwarfed in speed and accuracy. The object detection algorithm based on deep learning is an end-to-end processing method, as shown in Figure 2. It does not need to optimize multiple independent steps like the traditional method. It optimizes the whole detection system uniformly. It can adapt to various complex scenes (there is no need for sea–land segmentation in nearshore and port) and has very strong robustness. Therefore, in recent years, deep learning-based SAR ship detection algorithms have become a new research hotspot.

**Figure 2.** The differences between the CFAR-based detector and the deep learning-based detector.

The advantages and disadvantages of deep learning and traditional-based detection algorithms in SAR images can be proved by qualitative analysis or quantitative experiments.

The detection method based on CFAR has four main shortcomings through qualitative analysis. Firstly, CFAR needs to set the size of the protection window according to the size of the ship. It works well in the case of local uniform clutter in a single ship. If several ships with different sizes are close, the inconsistency between the change of ship size and the fixed protective window will lead to missed detection. Secondly, the CFAR algorithm needs to accurately model SAR images, which is difficult to implement. Thirdly, the essence of CFAR is an unsupervised algorithm, and its performance is essentially worse than the supervised algorithm (Faster R-CNN (region-based convolutional neural network), YOLO (you only look once), SSD (single shot detector), etc.). Fourthly, the CFAR algorithm and discrimination algorithm is a system pieced together after multiple links are debugged separately, and its performance cannot be compared with the end-to-end deep learning algorithm.

Sun Xian carried out a comparative experiment between the classical ship detection algorithm and the deep learning algorithm [65]. In the paper, the classical ship detection algorithms (optimal entropy automatic threshold method and CFAR method based on K distribution) are tested and analyzed on the AIR-SARShip-1.0 dataset. The experiment results are shown in Table 1. We can find that the performance of the deep learning algorithm is significantly better than that of the traditional algorithm.

**Table 1.** The performance difference between traditional-based detectors and deep learning-based detectors on the same condition [65].

Before SSDD, there are six papers that use convolutional neural networks [1–6] to detect ships in SAR images. We think that these six papers are not based on deep learning. The reasons are as follows. Firstly, some algorithms are not end-to-end, they just use CNN as a component in the traditional detection process. Secondly, although some algorithms

are end-to-end, the dataset and evaluation criteria are not public, which is difficult for future researchers to reproduce, and the results are also not comparable.

Due to the important role of SSDD, we take the publication date of SSDD paper as the time separation point between the traditional and deep learning-based detection algorithms. Therefore, we believe that ship detection in SAR images entered the era of deep learning on 1 December 2017, as shown in Figure 3. A large number of researchers gradually began to abandon the traditional detection algorithm based on CFAR and adopt the advanced detection method based on deep learning [6–182]. The overview of these deep learning-based detectors in SAR images is the focus of this paper.

**Figure 3.** The time divisions of past, present and future.

#### **4. Present—The Deep Learning-Based SAR Detection Algorithms**

*4.1. The General Overview of the 177 Papers*

#### 4.1.1. The Countries

In the country view, we can find that 90% of the papers' authors are Chinese, which is shown in Figure 4. There is no doubt that Chinese researchers have been the mainstream in this direction. Several public datasets are constructed by Chinese researchers, which further prove the above opinion.

**Figure 4.** The percentages of the author countries.

#### 4.1.2. Journal or Conference

A total of 63% of the 177 papers are published in journals, and 37% are in conferences. The most common journals and conferences are Remote Sensing and IEEE International Geoscience and Remote Sensing Symposium (IGARSS), respectively.

#### 4.1.3. Timeline of the 177 Papers

The timeline of the deep learning based SAR ship detectors is shown in Figure 5.

Gray lines and gray circles in the figure represent the time, the purple bar represents the number of papers in the current year, and the red circles represent the public time of the dataset. From the timeline, we can find that in the passing five years, the number of papers about deep learning-based SAR ship detectors becoming more and more. The period 2016–2017 is in the transitional period between traditional and deep learning methods, and there are sporadic papers. Additionally, due to the lack of a unified dataset and the lack of in-depth understanding of deep learning and computer vision algorithms, the application of deep learning algorithms is not thorough. This situation did not change until the emergence of the first dataset (SSDD) paper at the end of 2017. SSDD discloses the data and evaluation criteria it used, which lays the foundation for the rapid development of this field. Since then, the fast lane of research was opened, and a large number of papers were published. The milestones of this field are the several open-access datasets, which are shown in the red circle above. With the increase in available datasets, more and more researchers are paying attention to this field.

#### 4.1.4. The Datasets and Satellites That Are Used

The datasets that those papers used are shown in Table 2. We can see that SSDD is the most frequently used dataset for now. It was used 83 times, 62.4% percent of the total. Additionally, the usage of several public datasets shows a gradual upward trend.

**Table 2.** The datasets that are used in the past five years.

Before the first dataset paper was published in 2017, different researchers adopted different SAR images and indicators to test their detectors. Thus, the results of the papers are not comparable. This phenomenon is not beneficial for the development of this field. In order to overcome this, we constructed the first dataset SSDD and it is open to the public. Meanwhile, we provide another dataset called SSDD+, which shares the same images with SSDD but has an oriented bounding box. With the rapid development of deep learning-based computer vision algorithms after 2019, SSDD draws more attention to researchers. Zhang analyzed the usage situation of SSDD in paper [170]. From this paper, we can find that SSDD becomes the most popular dataset, though it has many drawbacks. In addition to this, SAR-Ship-Dataset and AIR-SARShip are showing great potential to be

the popular dataset. The other datasets were seldom used, as their public dates are a little late. As deep learning models need more data to prevent overfitting, the future of this field is to merge them into a large dataset and provide the benchmarks on the large dataset with the common detection algorithms in computer vision. We think that, if the dataset is big enough, the benchmark is whole enough, and the maintenance is regular enough, it will be accepted by most researchers. This is the focus of our future work.

Table 3 shows the SAR satellites that papers used besides the public dataset. We can find that SAR images from Sentinel-1 are the most frequently used all the time. This is because the data are easy to acquire, and can be downloaded for free.

**Table 3.** The satellites that are used in the paper beside the datasets.

However, as China's first C-band multi-polarization SAR satellite Gaofen-3 was officially put into use on 23 January 2017, the policy of obtaining Gaofen-3images has become easier and easier. More and more papers use Gaofen-3 as the source image.

#### 4.1.5. Deep Learning Framework

A deep learning framework can reduce the workload of researchers [183–187]. So, since the emergence of CAFFE (Convolution Architecture For Feature Extraction) [188] in 2017, it gets more and more attention from researchers. Table 4 shows the deep learning framework those 177 papers used. We can find that in the beginning years (2017–2018), CAFFE is the most frequently used framework. It is because CAFFE is the first common deep learning framework that researchers use and most of the detection algorithms in computer vision are based on CAFFE, for example, Faster R-CNN and SSD. In order to improve the efficiency of the deep learning framework, Google provided TensorFlow [189] in 2017. Compared with CAFFE, Tensorflow is more powerful and easier to use. A lot of researchers adopt Tensorflow as their framework. After Tensorflow, PyTorch [190] was promoted by Facebook FAIR in 2017. PyTorch is more suitable for researchers, and the number of users surpasses the Tensorflow gradually. In addition to CAFFE, Tensorflow and Pytorch, Keras, DarkNet and PaddlePaddle are also used by some researchers. Due to the fact that most detection algorithms in computer vision are based on Tensorflow and Pytorch, we recommend researchers in this area use them as the deep learning framework.

**Table 4.** The deep learning framework those papers used.

#### 4.1.6. Performance Evolution

Tables 5–11 show the performance of several public datasets. In the 'AP' column, the large number represents AP50, and the small one represents AP. AP50 refers to the average precision with IoU (intersection over union) = 50%. AP refers to the value of IoU from 50% to 95% in steps of 5% and then calculates the average value of AP under these IoUs. Normally, AP50 is higher than AP. AP50 and AP are usually used in PASCAL VOC and MS COCO, respectively. Since the dataset contains only one class of ships, the mAP value is the same as the AP value.

In the table, the italics represent the performance of two-stage detectors, and the nonitalics represent the performance of single-stage detectors. The blue, red, green, purple, and golden colors represent anchor-free, train from scratch, oriented bounding box, multi-scale, and attention detectors. The underlines represent real-time detectors.

The number of papers in Tables 5–11 is less than in Table 2. This is because some of the papers in Table 2 did not use the AP or AP50 as the evaluation indicator, so we do not show them in Tables 5–11.

From Table 5, we can see that there are 52 papers that are trained and tested on SSDD. Additionally, in the past five years, AP50 of detectors on SSDD boosted from 78.8% in 2017 to 97.8 % in 2022. The testing time is also getting faster and faster. What should be noticed is that as the train-test division is ambiguous in the original SSDD, so the AP in Table 5 is not comparable to some extent. That is also why we recommend the following researchers adopt the Improve SSDD [170] as the new standard.

**Table 5.** The performance evolution of detectors on SSDD (The data come from the 177 papers).

From Table 6, we can see that there are only four papers that are trained and tested on SSDD+, and the AP50 performance is increased from 84.2% in 2018 to 94.46% in 2021. The overall performance is a bit lower than that of SSDD. That is because the detectors on SSDD+ should predict an additional parameter (angle). We also find that the SSDD+ is seldom used compared with SSDD. That is, few researchers are interested in oriented bounding box detection in this area.

**Table 6.** The performance evolution of detectors on SSDD+ (the data come from the 177 papers).

From Table 7, we can see that there are 14 papers that are trained and tested on SAR-Ship-Dataset, and the AP50 performance is boosted from 89.07% in 2019 to 96.1% in 2021. The running speed is also accelerated to 60.4 FPS with 96.1% AP. The overall performance is a bit lower than that of SSDD. That is because this dataset is relatively larger than SSDD.

**Table 7.** The performance evolution of detectors on SAR-Ship-Dataset (the data come from the 177 papers).

From Table 8, we can see that there are only four papers are trained and tested on AIR-SARShip, and the AP50 performance is boosted from 88.01% in 2019 to 92.49% in 2021. In addition, the running speed becomes 7.98 times faster (from 41.6 ms to 5.22 ms). The overall performance is a bit lower than that of SSDD.

**Table 8.** The performance evolution of detectors on AIR-SARShip (the data come from the 177 papers).

From Table 9, we can see that there are only nine papers are trained and tested on HRSID, and the AP50 performance is boosted from 89.3% in 2019 to 94.4% in 2021. The overall performance is a bit lower than that on SSDD. That is because this dataset is relatively larger than SSDD.

**Table 9.** The performance evolution of detectors on HRSID (the data come from the 177 papers).

From Table 10, we can see that there are only three papers are trained and tested on LS-SSDD-v1.0, and the AP performance is boosted from 72.3% in 2019 to 75.5% in 2022. The overall performance is a bit lower than that on SSDD. LS-SSDD-v1.0 is specially used for large-scale SAR ship detection, which is fit for satellite-based SAR systems. It should be used more in the future.

**Table 10.** The performance evolution of detectors on LS-SSDD-v1.0 (the data come from the 177 papers).

The above datasets are relatively smaller than the datasets used in computer vision. In order to improve the generalization ability of the detector, researchers should use a large dataset. Some researchers merge several datasets into a large one as shown in Table 11. From Table 11, we can see that there are three papers that are trained and tested on the composite dataset, and the AP performance is 81.13%, 71.4%, and 95.1%, respectively. As deep learning-based detectors are data-hungry, we should merge the public datasets into a large one to prevent over-fitting.

**Table 11.** The performance evolution of detectors on other datasets (the data come from the 177 papers).

#### *4.2. The Algorithm Taxonomy of the 177 Papers*

We divide the 177 papers into 10 categories, they are papers about datasets, twostage detectors, single-stage detectors, anchor-free detectors, train from scratch detectors, detectors with the oriented bounding box, multi-scale detectors, detectors with attention module, real-time detectors, and others. The percentages of each algorithm are shown in Table 12. What should be explained is that the summation of the percentages is larger than 1. This is because many algorithms in the papers have several attributes. For example, it not only belongs to the single-stage detector but is also trained from scratch.

**Table 12.** The percentage of each algorithm.

From Table 12, we can find the following conclusions. Firstly, there are eight papers that introduce the datasets to the researchers. They make a great contribution to this field. Secondly, two-stage detectors used in this field are slightly more than single-stage detectors. This is partly because the two-stage detectors have higher accuracy than the single-stage detectors in most cases. In addition, accuracy is the first consideration at the moment. Thirdly, anchor-free detectors, detectors trained from scratch, oriented bounding box detectors, and detectors with attention modules almost have a percentage of 5–6% among the 177 papers. This is because, as the above four directions are rare, they are not yet noticed by many researchers. In fact, these directions can overcome the problems of the ship size distribution abnormal and the lack of SAR images. They should be paid more attention in the future. Fourthly, almost 14% of papers are about multi-scale SAR ship detection, which is a little higher than other directions. This is because, compared with objects in computer vision images, ships in SAR are rather small. In order to improve the performance, detectors should pay more attention to multi-scale ships. Fifthly, 14.20% of papers are classified as others, which represents that these papers do not belong to the nine categories. Sixthly, only three papers, which is 1.7% of the 177 papers are reviewed in this field. Considering the active research in this field, it is not enough for now. This is one of the motivations for our work.

#### *4.3. The Public Datasets*

#### 4.3.1. Overview

As far as we know, there are 10 public datasets that are used for training and detecting ships in SAR images. They are SSDD(SSDD+) [11], SAR-Ship-Dataset [38], AIR-SARShip1.0 [65], HRSID [94], LS-SSDD-v1.0 [101], AIR-SARShip2.0 [191], Official-SSDD [170], SRSDD-v1.0 [177] and RSDD-SAR [192]. Table 13 shows the detailed information of the 10 public datasets, in which the annotations of SSDD+, Official-SSDD, SRSDD-v1.0, and RSDD-SAR are the oriented bounding box.

**Table 13.** Detail information of existing public datasets.

In addition to these, SMCDD [182] is a good dataset based on China's first commercial SAR satellite HISEA-1. It has 1851 bridges, 39,858 ships, 12,319 oil tanks, and 6368 aircraft. It shows a great advantage in multi-class ship detection.

In the future, it is very necessary to combine the above datasets into a large one to avoid the problem of overfitting.

In the following part, we will introduce the details of the datasets and evaluate their advantages and drawbacks.

#### 4.3.2. SSDD, SSDD+ and Official-SSDD

We made our dataset SSDD publicly available at the conference of 2017BIGSARDATA in Beijing [11]. SSDD is the first open dataset in this community. It can be a benchmark for researchers to train and evaluate their algorithms.  The ships in SSDD have rich diversity, including small-size ships, complex backgrounds, and dense arrangements near the wharf. We also give the statistical results of the length, width, and aspect ratio of the ship bounding box in SSDD. The papers that used SSDD and their performance are shown in Table 5.

At the same time, based on 1160 SAR images of SSDD, we use the oriented bounding box to relabel the ship and obtain the dataset SSDD+. SSDD+ is the first dataset for SAR ship detection with an oriented bounding box. The papers that used SSDD+ and their performance are shown in Table 6.

At that time, there were some problems in SSDD due to the lack of understanding of computer vision and deep learning. The drawbacks of SSDD are the coarse annotations and ambiguous standards of use. It hinders fair comparisons and effective academic exchanges in this field.

In September 2021, Zhang [170] systematically analyzed and improved the problem of SSDD; they call it Official-SSDD. Zhang relabeled ships in SSDD and proposed three new datasets; they are bounding box SSDD, rotatable bounding box SSDD, and polygon segmentation SSDD. In addition, they also formulate some standards: the train-test division, the inshore-offshore protocol, the ship-size definition, the determination of the densely distributed small ship ships, and the determination of the densely parallel berthing at ports ship samples. We suggest that follow-up researchers use the Official-SSDD and standards proposed in paper [170] to carry out their relevant research.

#### 4.3.3. SAR-Ship-Dataset

The training of the deep learning model depends on a large amount of data, and the amount of SSDD is relatively small. To solve this problem, Wang Chao [38] constructed a dataset called SAR-Ship-Dataset. SAR-Ship-Dataset contains 43,819 images and 59,535 ships, which are more than SSDD. The sources of SAR-Ship-Dataset are 102 Gaofen-3 images and 108 Sentinel-1 SAR images. These ships have distinct scales and backgrounds. The resolution, incident angle, polarization mode, and imaging mode are also diverse, which are helpful for the deep learning models to fit different conditions. The papers that used SAR-Ship-Dataset and their performance are shown in Table 7.

#### 4.3.4. AIR-SARShip

A dataset containing more diverse scenes and covering various types of ships will help to train a model with better performance, stronger robustness, and higher practicability. In order to achieve the above purpose, Sun Xian constructed a dataset based on the Gaofen-3 satellite, named AIR-SARShip-1.0 [65].

It contains a total of 31 large images. A total of 21 images are training data and the other 10 images are testing data. The image resolutions include 1 m and 3 m. The image size is about 3000 × 3000 pixels. The information of each image in the dataset includes image number, pixel size, resolution, sea state, scene, and the number of ships. The dataset has the characteristics of a large scene and a small ship.

On the basis of version 1.0, Sun Xian and other researchers added more Gaofen-3 data to build AIR-SARShip-2.0. The dataset contains 300 SAR images. The scene types include ports, islands, reefs, sea surfaces with different levels of sea conditions, etc. The annotation information includes the location of ships, which has been confirmed by professional interpreters.

The papers that used AIR-SARShip and their performance are shown in Table 8.

#### 4.3.5. HRSID

The original SAR image used to construct HRSID [94] includes 99 Sentinel-1B images, 36 TerraSAR-X images, and 1 TanDEM-X image. HRSID has 5604 high-resolution SAR images with 800 × 800 pixels to meet the needs of actual training for GPU. It is designed for ship detection and segmentation based on CNN, and it only contains one category of ships. It is divided into 65% training set and 35% testing set. It uses polygons to label the ship. In order to reduce the deviation of the ship detection algorithm, the interference derived from the ship is marked as a part of the ship.

According to statistics, the total number of ships marked in HRSID is 16,951, and each SAR image contains an average of three ships. The number of small ships, medium ships, and large ships accounted for 54.5%, 43.5%, and 2% of all ships, respectively. The bounding box areas of small ships, medium ships, and large ships account for more than 0~0.16%, 0.16~1.5%, and 1.5% of SAR images, respectively. Therefore, ships are sparsely distributed in SAR images.

The papers that used HRSID and their performance are shown in Table 9.

#### 4.3.6. LS-SSDD-v1.0

Zhang Xiaoling [101] constructed the SAR ship detection dataset LS-SSDD-v1.0 with a large scene and small ships. The dataset consists of 15 pieces with a size of 24,000 × 16,000 pixels Sentinel-1 SAR images. Each image is directly divided into 600 sub-images with 800 × 800 pixels. The dataset contains 6015 ships. LS-SSDD-v1 can support researchers to flexibly apply the dataset. The optical information provided in Google Earth software and ship information provided by AIS is used for the annotation of LS-SSDD-v1.0. The coastline of the imaging area in the dataset is relatively complex, the land area is smaller than the ocean area, and the ships in the inland river are densely distributed. The dataset has the following characteristics: contains large scenes, focus on the small ships, rich pure background, etc. It also provides a large number of performance benchmarks of detection algorithms on datasets.

The papers that used LS-SSDD-v1.0 and their performance are shown in Table 10.

#### 4.3.7. SRSDD-v1.0

The original images of SRSDD-v1.0 are from Gaofen-3 [177]. It contains 30 panoramic SAR images of port areas. It is annotated with an oriented bounding box. Optical images (Google Earth or GF-2) are used to assist the annotation. The image size is set to 1024 × 1024. The annotation format is the same as DOTA. The coordinates of the four corners of the box, the category, and whether it is difficult to identify are given in annotation files.

It contains 666 images. A total of 420 images with 2275 ships include the land cover. A total of 246 images with 609 ships only contain the sea in the background. It has six categories: ore-oil ships (166), bulk cargo ships (2053), fishing boats (288), law enforcement ships (25), dredger ships (263), and container ships (89). The dataset has a certain data imbalance problem.

#### 4.3.8. RSDD-SAR

The RSDD-SAR dataset consists of 84 scenes of Gaofen-3 and 41 scenes of TerraSAR-X. RSDD-SAR has 7000 images, including 10,263 ships, of which 5000 are randomly selected as the training set and the other 2000 as the testing set. By analyzing the distribution of ship angle and aspect ratio in the dataset, it can be found that the angle of ships in the dataset is evenly distributed between 0◦ and 180◦, and the aspect ratio is concentrated between two

and six. It indicates that the dataset has the characteristics of arbitrary rotation direction and a large aspect ratio. The dataset has the characteristics of a high proportion of small ships, which can be used to verify the performance of a small ship detection algorithm. The RSDD-SAR dataset contains vast sea areas, ports, docks, waterways, and other scenes with different resolutions, which are suitable for practical applications.

#### *4.4. Two-Stage Detectors*

The deep learning-based object detection algorithm can be divided into single-stage detectors and two-stage detectors. The single-stage detectors use a full convolution network to classify and regress these anchor boxes once to obtain the detection results. The two-stage detectors use a CNN to classify and regress these anchor boxes twice to obtain the detection results. The principles of single-stage and two-stage detection algorithms are shown in Figure 6.

**Figure 6.** The principle of single-stage and two-stage detectors.

Classical two-stage detectors are Faster R-CNN, R-FCN (fully convolutional network) [193], feature pyramid networks (FPN) [194], Cascade R-CNN [195], Mask R-CNN [196], and so on [197]. Faster R-CNN is the foundation work, and most of the two-stage detectors are improved based on it.

Among the 177 papers, most of the papers are improved from the following aspects: backbone network, region proposal network (RPN), anchor box, loss function, and nonmaximum suppressing (NMS). They are shown in Figure 7. Compared with computer vision, the research in this field lags behind, and other more advanced two-stage detection algorithms have not been used here.

**Figure 7.** The two-stage SAR ship detectors.

#### 4.4.1. Backbone Network

There are three main directions in the improvement of the backbone network, namely FPN, feature fusion, and attention.

FPN produces a feature pyramid structure that combines low-resolution, which has strong semantic features with high-resolution, which has weak semantic features. It includes a bottom-up channel, a top-down channel, and a skipping connection. It predicts independently at all levels, which only brings minimal additional calculation and storage consumption. It improves the detection result of small-size ships and thus is widely used. A lot of work has been completed to improve FPN in the computer vision field, such as ASFF [198], NAS-FPN [199], and BiFPN [200].

There are six papers [42,62,63,69,91,110] adopted and improved FPN in this field. Cui et al. [42] proposed a DAPN (dense attention pyramid network) structure. It densely connects the convolution block attention module from the top to the bottom of the pyramid network. By this, rich features including resolution and semantic information are extracted for multi-scale ship detection. Li et al. [62] used a convolution block attention module (CBAM) [201] to control the degree of upper- and lower-level feature fusion in FPN. Liu et al. [63] proposed a scale-transferrable pyramid network. It densely connects each feature map from top-to-down using scale-transfer layer. It can expand the resolution of feature maps, which is helpful for detection. Wei et al. [69] adopted a parallel high-resolution feature pyramid network to make full use of the feature mapping of high-resolution and low-resolution convolution for SAR ship detection. Zhao et al. [91] adopted receptive fields block and convolutional block attention module to build a top-down fine-grained feature pyramid. It can capture features of ships with large aspect ratios and enhance local features with their global dependences. Hu et al. [110] used a dense connection to a feature pyramid network, in which the shallow features and deep features are processed differently. It considered the differences between different levels.

There are three papers [11,57,115] that improved the backbone network through feature fusion. Li et al. [11] fused the feature maps from convolutional layer 3 to layer 5. The fusion includes the normalization and 1 × 1 convolution. Normalizing each RoI (region of interest) pooling tensor can reduce the scale differences between the following layers. It can prevent the 'larger' features from dominating the 'smaller' ones and make the algorithm more robust. This modification stabilizes the system and increases the accuracy. Yue et al. [57] fused the semantically strong features with the low-level highresolution features, which is helpful for reducing false alarms. Li et al. [115] presented a jump connection structure to extract the features of each scale target in the SAR image. It can improve the ability of recognition and localization.

There are five papers [17,29,62,122,137] that improve the backbone network through the attention module (SENet). It squeezes the feature map along the space and the channel direction, which can explicitly model the interdependence between feature channels, and then automatically obtain the importance of each feature channel through learning. It can improve the useful features and suppress the features that are not useful for the current task according to the importance.

#### 4.4.2. RPN

Another direction is improving the RPN module of Faster R-CNN. Paper [16,25,34,46,87] did not use a single feature map to generate proposals but generated proposals from each fused feature map. Liu et al. [36] designed a scale-independent proposal generation module, which extracts the features such as edge, super-pixel, and strong scattering component from SAR image to obtain ship proposals, and sorts whether the proposals contain ships from the integrity and tightness of the contour. In paper [160], candidate proposals are extracted from the original SAR image and the denoised SAR image, respectively, and then combined to reduce the impact of noise in the SAR image on ship detection. They can improve the performance of multi-size ships to some extent.

#### 4.4.3. Loss Function

Faster R-CNN forces the ratio of positive and negative samples to 1:3 to solve the problem of unbalanced positive and negative proposals. Similar work in the field of computer vision includes focal loss, OHEM (online hard example mining) [202], GHM (gradient harmonizing mechanism) [203], and Libra R-CNN [204]. The paper [16,78], respectively, adopted focal loss to increase the weight of hard negative samples and reduce the weight of simple samples, so as to avoid the problem that a large number of simple samples cover a small number of hard negative samples in the training process.

#### 4.4.4. Anchor and NMS

Faster R-CNN uses three scales and three aspect ratios, producing a total of 60 × 60 × 9 anchor boxes. However, the ship size in the SAR image is extremely small and sparse. There will be a waste in using dense anchor boxes for ship detection in SAR images. Yue et al. [57] and Wang et al. [122] set the parameters of the anchor box based on the analysis of the actual size and distribution of the ship, mainly reducing the size of the anchor box and selecting the appropriate shape. Chen et al. [70] and Li et al. [115] used K-means to obtain the distribution of the ship size, so as to obtain the appropriate anchor box and reduce the difficulty of learning.

Wei et al. [69] and Wang et al. [78] used soft NMS [205] to replace NMS. Soft NMS improves the discrimination process of IoU and threshold in the cycle process and uses weights to attenuate scores to avoid accuracy loss.

#### 4.4.5. Others

ISASDNet (instance segmentation assisted ship detection network) was proposed based on Mask R-CNN in the paper [163]. It has two branches: detection and segmentation. The two branches output interaction to improve the detection results. Gui et al. [34] proposed a lightweight detection head with a large separable convolution kernel and position-sensitive pooling, which improves the detection speed.

#### *4.5. Single-Stage Detectors*

The two-stage detectors generate a candidate box first and then identify and regress the candidate box, which is quite different from the principle of human eyes. The singlestage detectors only need to look at the picture once and can predict what the object is and where the object is. It is similar to the human eyes. In addition, they are quite faster than two-stage detectors.

Classical single-stage detectors are YOLO, SSD, RetinaNet [206], and CornerNet [207]. YOLO and SSD are the two most popular single-stage detection algorithms, and most of the subsequent single-stage works are based on them.

The single-stage ship detectors in SAR images are shown in Figure 8.

**Figure 8.** The single-stage SAR ship detectors.

#### 4.5.1. YOLO and SSD Series in Computer Vision

YOLOv1 [186] regards object detection as a regression problem, and it outputs the spatially separated bounding box and related class probability simultaneously. A neural network can predict the bounding box and class probability from the image in one forward calculation. The speed is very fast, but it has an inaccurate location prediction, and the recall is low. YOLOv2 [208] uses the multi-scale training method. It predicts the offset rather than the parameter itself. The offset value is slightly smaller, which can increase the accuracy of prediction. It uses an anchor mechanism to obtain anchor box parameters by clustering the object size in the dataset. The backbone network adopts DarkNet-19. Although the detection head has changed from 7 × 7 to 13 × 13, the detection result of a small object is still poor. The YOLOv3 [209] detection head includes three branches: 13 × 13, 26 × 26, and 52 × 52, which can take into account large, medium, and small objects and make the location prediction more accurate. The anchor mechanism of YOLOv3 is the same as that of YOLOv2. YOLOv4 [210] uses two anchors for one ground truth, while YOLOv3 uses only one anchor for one ground truth. With this, the problem of imbalance between positive and negative samples is alleviated. CIoU loss is adopted to solve the problems of MSE (mean squared error) loss, IoU loss, GIoU, and DIoU [211–214]. YOLOv4 also uses several techniques to achieve state-of-the-art results. YOLOv5 adopts adaptive anchors and uses the network to learn anchor parameters. Its detection head is the same as YOLOv3 and YOLOv4. It is slightly weaker than YOLOv4 in performance, but much faster than YOLOv4, and has strong advantages in the rapid deployment of the model.

SSD detection algorithm combines the regression idea with the anchor box (default frame) mechanism. It eliminates the candidate region generation and subsequent pixel or feature resampling stage (RoI pooling) in the two-stage algorithm. It encapsulates all calculations in one network, making it easy to train and very fast. RFBNet [215] and M2Det [216] are two successors of SSD. They use receptive fields and multi-level feature pyramid networks to improve the classical SSD, respectively.

Single-stage SAR ship detection algorithms can be divided into three categories: SAR image ship detection based on the YOLO series, SAR ship detection based on the SSD series, and other algorithms.

#### 4.5.2. SAR Ship Detection Based on YOLO Series

YOLO series are widely used in this field. The improvements mainly focus on lightweight backbone network designing, multi-layer feature fusion, anchor box generation, multi-feature map prediction, loss function, etc.

YOLOv2. Deng et al. [33] and Chang et al. [39] adopted YOLOv2 to detect ships in SAR images. Paper [39] proposed YOLOv2-reduced which reduces some layers of YOLOv2. YOLOv2-reduced has an AP of 89.76% with 10.937 ms and 44.72 BFLOPS compared with YOLOv2, which has an AP of 90.05% with 25.767 ms and 50.17 BFLOPS.

YOLOv3. Zhang et al. [82] accelerated the original YOLOv3 by using DarkNet-19 as the backbone network. Additionally, it reduces the repeated YOLOv3-scale1, YOLOv3-scale2, and YOLOv3-scale3. Zhu et al. [116], Chaudhary et al. [123], and Jiang et al. [135] used the classical YOLOv3 with some techniques to detect ships in SAR images. Wang et al. [119] proposed SSS-YOLO which redesigned the feature extractor network to enhance the spatial and semantic information of small ships. It adopts a PAFN (path argumentation fusion network) to fuse different features in a top-down and bottom-up manner. SSS-YOLO has a better performance for small ships in SAR images. Hong et al. [158] improved the performance of YOLOv3 with some techniques. The improved clustering algorithm K-means++ generates an anchor box, which improves the performance of YOLOv3 for multi-scale ships. The Gaussian parameter for ship detection is introduced to add an uncertainty estimator for the positioning of the bounding box. Four anchor boxes are assigned to each detection scale instead of three in YOLOv3. Zhang et al. [40] used the idea of the YOLO algorithm, the input image meshes, and the depth separable convolution is used to improve the detection speed. MobileNet is used as the feature extractor to detect

ships under three scales: 13 × 13, 26 × 26, and 52 × 52. The size of the anchor box can be obtained by the K-means algorithm. D-CNN-13 has a big receptive field with anchor box widths and heights of (9, 11), (11, 22) (14, 26). D-CNN-26 has a medium receptive field with anchor box widths and heights of (16, 40), (17, 12) (27, 57). D-CNN-52 has a small receptive field with anchor box widths and heights of (28, 17), (57, 28) (69, 72). Zhang et al. [54] used depth convolution and point convolution to replace the traditional convolution neural network, and adopt a multi-scale detection mechanism, concatenation mechanism, and anchor box mechanism to improve the detection speed. The detection network is composed of three parts, which means that it can detect an input SAR image under three different scales (5 × 5, 10 × 10, and 20 × 20), and then obtain the final ship detection results. It has nine anchor boxes for three detection scales, so it can detect up to nine ships in the same grid cell. Zhou et al. [102] designed a CNN named LiraNet, which has low complexity, few parameters, and a strong feature representation ability. LiraNet combines the idea of dense connections, residual connections, and group convolution, and it includes stem blocks and extractor modules. The network is the feature extractor of Libra-YOLO. Lira-YOLO has only 2.980 Bflops, and the parameter quantity is only 4.3 MB. It has good accuracy with less memory and computational cost compared with tiny-YOLOv3. In [151], DarkNet-53 with the residual unit is used as the backbone to extract features, and a top-down pyramid structure is added for multi-scale feature fusion. Soft NMS, mix-up, mosaic data augmentation, multi-scale training, and hybrid optimization are used to boost the performance. The 13 × 13, 26 × 26, 52 × 52 feature maps with the large, medium, and small receptive fields are responsible for large, medium, and small ships, respectively. The model is trained from scratch to avoid the learning objective bias of pre-training. The detection speed is fast, about 72 frames per second.

YOLOv4. Ma et al. [156] proposed YOLOv4-light, which is tailored to reduce the model size, detection time, number of computational parameters, and memory consumption. The three-channel images are used for compensating for the loss of accuracy. Liu et al. [181] proposed a detection method based on YOLOv4-Lit [217], whose backbone is MobileNetv2. A receptive field block is used for multi-scale target detection. It has an AP of 95.03% with 47.16 FPS and 49.34 M model size.

YOLOv5. Tang et al. [144] proposed N-YOLO based on YOLOv5. N-YOLO adopts a noise level classifier to classify the noise level of SAR images. SAR ship potential area extraction module is used to extract the complete region of potential ships. Zhou et al. [179] proposed a multi-scale ship detection network based on YOLOv5. It has the cross-stage partial network to improve feature representation capability, and the feature pyramid network with fusion coefficients module to fuse feature maps adaptively. It has a good tradeoff between model size and inference time.

Others. Zhang et al. [84] proposed ShipDeNet-20. It has only 20 convolution layers, and the model size is smaller than 1 MB, which is lighter than the other state-of-the-art detectors. ShipDeNet-20 is based on YOLO and is trained from scratch. Feature fusion module, feature enhance module, and scale share feature pyramid module are proposed to make up the accuracy loss of the raw ShipDeNet-20. It has a good tradeoff between accuracy and speed. Zhu et al. [175] proposed DB-YOLO. It is composed of a feature extraction network, duplicate bilateral feature pyramid network, and detection network. The single-stage network can meet the requirements of real-time detection, and it uses cross-stage partial to reduce redundant parameters. A duplicate bilateral feature pyramid network can enhance the fusion of semantic and spatial information. It alleviates the problem of small ship detection. CIoU loss is used as the loss function, as it has a faster convergence speed and better performance.

#### 4.5.3. SAR Ship Detection Based on SSD Series

Wang et al. [14,18] directly used SSD and do not improve it. Papers [51,98,108] are the detection algorithms trained from scratch based on SSD. Most of the other papers improve the backbone network of SSD to make the model have a stronger feature extraction ability.

Chen et al. [15] adopted a two-stage regression network based on SSD to improve the performance of small ships, namely R2RN (robust two-stage regression network). R2RN connected an anchor modified module and object detection module to inherit the essence of the feature pyramid. Ma et al. [30] proposed an SSD model with multi-resolution input, which can extract richer features. Papers [43,44] applied the attention mechanism to SSD and design a new loss function based on GIoU. Li et al. [47] analyzed the reasons for the low detection accuracy of small and medium-sized ships in SSD and puts forward improvement strategies. Firstly, the anchor box optimization method based on K-means clustering is adopted to improve the matching performance of the anchor box. Secondly, a feature fusion method based on deconvolution is proposed to improve the representation ability of the underlying feature map. Chen et al. [55] adopted the attention mechanism and multi-level features to improve the feature extraction ability of the backbone network. Han et al. [99] used deconvolution to enhance the representation of small ships in the pyramid and improved the detection accuracy of SSD. Zhang et al. [113] token the original SAR image and saliency map as the input and fused the fusion of their features to reduce the computational complexity and network parameters. Chen et al. [114] proposed SSDv2, which adds a deconvolution module and prediction module on the basis of SSD to improve the detection accuracy. Jin et al. [149] improved SSD by feature fusion and squeezeexcitation module.

Sun et al. [162] proposed SANet (semantic attention-based network). It combines semantic attention, focal loss, label, and anchor assigning to improve the performance without increasing computation. Papers [104,159] adopted M2Det to detect ships in SAR images.

#### 4.5.4. Others

RefineDet adopts a two-step cascade regression strategy to predict the position and size of objects. It can make the single-stage detectors obtain the accuracy of the twostage detector without increasing computation. It is widely used in computer vision. Zhu et al. [159] adopted RefineDet to detect ships in SAR Images, which achieve an AP of 98.4%. In [169], GHM was used as the loss function of RefineDet, so that the network can make full use of all examples, and adaptively increase the weight of difficult cases. A multi-scale feature attention module is added to the network to highlight important information and suppress the interference caused by clutter. It achieves 96.61% precision on AIR-SARShip-1.0.

#### *4.6. Anchor Free Detectors*

#### 4.6.1. Development of Anchor Free Detection Algorithm in Computer Vision

The anchor box is the key to the success of Faster R-CNN and SSD. The backbone network extracts features from the input image to obtain the feature map, and each pixel on the map is the anchor point. Taking each anchor point as the central point and artificially setting different scales and aspect ratios, multiple anchor boxes can be obtained. Anchor box has the following two advantages: firstly, it can generate dense candidate boxes, which is convenient for the network to classify and regress the targets. Secondly, it can improve the recall ability and is suitable for small target detection.

However, the anchor box needs to be designed manually by experience, which has the following defects: firstly, hyper-parameters need to be set, such as the number, size, aspect ratio, IoU threshold, etc. Secondly, in order to match the ground box, a large number of anchor boxes need to be generated, which are computationally intensive. Thirdly, most of them are invalid, which will lead to an imbalance between positive and negative samples. Fourthly, it is necessary to adjust the anchor box according to the size and shape distribution of the dataset.

The anchor-free detector opens up another idea by eliminating the predefined anchor box. It can directly predict several key points of the target from the feature map. For example, CornerNet, ExtremeNet [218], CenterNet [219], Objects as Points [220], FCOS (fully convolutional one-stage) [221] and FoveaBox [222].

The anchor-free detectors can avoid various problems and has great application potential in SAR ship detection. For example, due to the small size and sparse distribution of ships, most of the candidate anchor boxes are invalid negative samples. The anchor-free detectors can neglect the invalid anchors and reduce the amount of the predicted boxes, thus improving the accuracy and speed simultaneously. The anchor-free ship detectors in SAR images are shown in Figure 9.

**Figure 9.** The anchor-free SAR ship detectors.

4.6.2. Development of Anchor-Free SAR Ship Detection Algorithm

Mao et al. [81] proposed a simplified U-Net [223] based anchor-free detector in SAR images. It includes ship bounding boxes regression network and score map regression network. The former is expected to be regressed based on each pixel in the input image. The latter is designed to predict a 2D probability distribution in which each score at each position indicates the likelihood of the current position in the center of any ship. Cui et al. [89] proposed a CenterNet (objects as points) based SAR ship detector. It predicts the center point of the target through key point estimation and uses the image information of the center point to obtain the size and position of the ship. There is no need to set anchors in advance and NMS is not needed, which greatly reduces the number of network parameters and calculations. Anchor mismatching of small ships is also reduced. Spatial shuffle-group enhance attention modules are used to extract features with more semantic information. Fu et al. [95] proposed an attention-guided balanced pyramid based on FCOS to improve the performance of small ships. Zhou et al. [97] proposed an anchor-free detector with dense attention feature aggregation. A lightweight feature extractor and dense attention feature aggregation are used to extract multi-scale features. A center-point-based ship predictor is used to regress the centers and sizes. There is no pre-set anchor and NMS, and thus the computational efficiency is high. Mao et al. [103] proposed a lightweight named ResSARNet with only 0.69 M parameters, and improved FCOS in four aspects: center-ness in bounding box regression branch, not in classification regression branch, center sampling, GIoU loss, and adaptive training sample selection. The network only needs 1.17 M parameters and can achieve 61.5% AP and 70.9% AR. An et al. [129] designed an anchor-free rotatable detector. It designs center point-scale and angle prediction to convert the conventional rotatable prior box mechanism into the center point-scale and angle prediction. The training procedure includes positive sample selection, feature encoding, and loss function designing. Wang et al. [133] proposed a CenterNet-based detector in SAR images. The spatial group-wise enhanced attention module is used to extract more semantic features.

Sun et al. [167] proposed category-position FCOS. The category-position module is used to optimize the position regression branch in the FCOS network. The classification and regression branches are redesigned to alleviate the imbalance between positive and negative samples during training. Zhu et al. [180] adopted FCOS as the base model to reduce the effect of anchors. A new sample definition method is used to replace the IoU threshold according to the differences between SAR images and natural images. The same resolution feature convolution module, multi-resolution feature fusion module, and feature pyramid module are used to extract features. The focal loss and CIoU are used to improve the performance further.

In all, researchers in SAR ship detection realize the benefit of anchor-free detectors. Additionally, more and more papers are appearing in this field. However, there is still a problem: the innovation is relatively weak and some of the existing achievements of computer vision are not used in this field.

#### *4.7. Detectors Trained from Scratch*

At present, most SAR image detector backbone needs to pre-train on the classification dataset of natural images, and then fine-tune on the ship detection dataset of SAR images (for example SSDD). This transfer learning can make the detection algorithm initialize better and make up for the problem of insufficient samples. However, there will be the following problems: firstly, there is learning bias. The loss function and category distribution between classification and detection are contradictory in essence. The models trained on classification are not fit for detection. Secondly, most backbone networks will produce a high receptive field through multiple down sampling in the latter layers, which is good for the classification but is harmful for the location. Thirdly, the pre-trained backbone networks are redundant and cannot be modified, which hinders the researchers to design CNN flexibly according to their needs.

In order to solve the problems of transfer learning, algorithms trained from scratch are proposed in computer vision, for example, DSOD (deeply supervised object detectors), DetNet, ScratchDet, and so on [224–227].