To restrict the search area, i.e., area processed by the detector components, a novel *sub-division-RPN layer* is applied on the output of *conv3\_3*¹, which is used as feature map for the RPN. The sub-division-RPN layer divides the output of *conv3\_3* into tiles of size 56 × 56 × 512. The offsets for cropping the tiles are set in a way so that the corresponding input image regions are equivalent for both the sub-division-RPN layer and sub-division-SAR layer. Besides the output of *conv3\_3*, the sub-division-RPN layer takes as additional input the classification results of the SAR classifier. Based on the classification scores, tiles that are likely to contain at least one object are passed to the RPN, while all other tiles are filtered out. Note that the passed tiles are stacked together into one batch, so that the RPN can generate region proposals for all relevant tiles at once.

¹ To adapt the number of channels required for the fully connected layers, a 1 × 1 convolutional layer is applied on the output of *conv3\_3* prior to the *sub-division-RPN layer*.

The original proposal layer computes the actual coordinates of each region proposal by adding the predicted offsets to the respective reference bounding box. As the reference bounding box is given in tile coordinates, the proposal layer of the RPN is modified in order to map the region proposals to their position in the input image. Therefore, a position vector generated by the sub-division-RPN layer is taken as an additional input. The position vector encodes the position of each tile that is forwarded from the sub-division-RPN with regard to its position in the input image. The actual coordinates of each region proposal are then computed by adding the offsets for the respective tile to its position given in tile coordinates. All mapped region proposals are then forwarded to the classification stage.

#### **7.2.2 Implementation Details**

To train the extended Faster R-CNN, a training strategy comprised of two stages is performed. Note that training both Faster R-CNN and the SAR classifier at once is not possible, as Faster R-CNN requires at least one GT object per training image, whereas the SAR classifier requires images with and without GT objects. In the first stage, Faster R-CNN is trained end-to-end for 60,000 iterations. For this, the training settings described in Section 5.1.3 are adopted. Then, the SAR classifier is trained in the second stage. To account for the input image dimensions of the default VGG16 classifier used as base architecture, the images of the DLR 3K dataset are split into sub-images of size 224 × 224 pixels. Adjacent sub-images exhibit an overlap of 46 pixels. The sub-images are divided into two categories: the first comprises all sub-images without any GT annotation and the second contains all sub-images with at least one GT annotation. For each sub-image, only GT annotations with an overlapping area of 10 or more pixels to the current sub-image are considered. Furthermore, data augmentation, i.e., horizontal and vertical flipping, is performed for each sub-image yielding a total of 19,200 training images, whereof 4,656 images belong to the category with at least one object. Example images of the training set are depicted in Figure 7.8.

**Figure 7.8:** Example images of the training set used to train the SAR classifier for DLR 3K. The top row shows examples for the category with at least one object, while the bottom row shows examples for the category without objects.

The SAR classifier is trained for 10,000 iterations with a batch size of 8. The initial learning rate is set to 0.001 and decreased after 5,000 iterations by a factor of 10. Note that all weights of Faster R-CNN are kept fixed in the second training stage.

For deployment, the sub-division-SAR layer, sub-division-RPN layer, and the modified proposal layer are added to the framework to perform the search area reduction and the detection in a single network as depicted in Figure 7.7. For each tile, NMS is applied on the 400 region proposals exhibiting the highest confidence score. The top-80 ranked proposals after NMS are then forwarded to the classification stage. As the input images are divided into 25 tiles, the maximum number of proposals before and after NMS are 10,000 and 2,000, respectively, which is equivalent to the numbers for Faster R-CNN without the SAR module (see Section 5.1.3). The further settings are analogous to the settings employed for Faster R-CNN without the SAR module.

#### **7.2.3 Ablation Experiments**

The impact of the proposed SAR module on the inference time is evaluated on the DLR 3K dataset and the VEDAI LCI dataset in order to account for different area categories with varying vehicle distributions, as the speed-up in inference time depends on the number of filtered out image regions. In case of the DLR 3K test set, 36.4% of the image tiles include at least one object, since DLR 3K comprising mainly urban and residential areas exhibits a high traffic volume. In contrast, VEDAI comprises more rural and nature areas with a low traffic volume, so that only 8.2% of the image tiles in the VEDAI test set contain at least one object.

**Table 7.16:** Comparison of the inference time for Faster R-CNN with and without SAR on the DLR 3K dataset. The overall inference time and in particular the inference time for the RPN and the classification stage are reduced by applying the SAR module.

Comparison of the inference time for Faster R-CNN with and without the SAR module on the DLR 3K dataset is given in Table 7.16. For this, all timings are performed analogous to the timings in Section 5.2.4 on two different devices. The SAR module aims at restricting the search area and consequently reducing the inference time without worsening the detection accuracy. Preliminary experiments showed that the best trade-off between runtime and detection accuracy is achieved for a confidence score threshold of 0.5 used to accept tiles for the RPN and the subsequent classification stage. Using higher threshold values results in false negative detections due to vehicles present in tiles that are filtered out. Thus, the threshold value is set to 0.5 in the following.

The inference time for the RPN is reduced by approximately 34% and 25% for the server and desktop setup, respectively, due to the clearly reduced number of feature map locations used to predict candidate regions and the clearly reduced number of region proposals that have to be processed in the subsequent proposal layer. The inference time for the classification stage is even reduced by about 83% and 73%, respectively, as the number of candidate regions to classify is considerably reduced. However, the overall inference time is only reduced by 21% and 16%, respectively, because of the auxiliary components of the SAR module and the inherent computational costs for its feature extraction, which takes about 27% of the overall inference time. The additional costs are mainly due to the cropping and reorganization of convolutional features in the sub-division-SAR and sub-division-RPN layer. Note that computational costs for the sub-division-RPN layer depend on the number of tiles considered for detection, as only the corresponding convolutional features are rearranged. Furthermore, additional costs depending on the number of tiles considered for detection result from the auxiliary computational operations in the proposal layer.

**Table 7.17:** Comparison of the inference time for Faster R-CNN with and without SAR on the VEDAI dataset. The overall inference time and in particular the inference time for the RPN and the classification stage are reduced by applying the SAR module.

The impact of the SAR module on the inference time for VEDAI is given in Table 7.17. The overall inference time is reduced by 51% and 42% on the server and desktop setup, respectively, while the AP only marginally decreases from 97.4% to 97.3%. The feature extraction for VEDAI is more time-consuming due to the larger input image dimensions, i.e., 1024 × 1024 vs. 936 × 936 pixels in case of DLR 3K. The speed-up is more distinctive compared to DLR 3K due to the lower traffic volume and consequently reduced number of tiles considered for the RPN and the classification stage. In particular, the inference time for the RPN is notably reduced, i.e., by 81% and 80%, respectively.

**Figure 7.9:** Classification results of the SAR module on DLR 3K. Highlighted regions are classified as region with at least one object and are considered for detection. Regions labeled in blue contain vehicles, whereas regions labeled in red contain no vehicles.

Qualitative visualizations of the reduced search area are depicted in Figure 7.9 and Figure 7.10. Highlighted regions possess a confidence score about the presence of at least one vehicle above 0.5 and are considered for detection. Regions labeled in blue are correctly classified and contain at least one vehicle, whereas regions labeled in red contain no vehicle. On both datasets, the SAR module is able to reliably identify tiles that contain at least one object even in case of housing areas with complex backgrounds and in case of areas off paved roads that typically contain no vehicles. Tiles without vehicles that are misclassified generally comprise structures with shapes similar to vehicles.

**Figure 7.10:** Classification results of the SAR module on VEDAI. Highlighted regions are classified as region with at least one object and are considered for detection. Regions labeled in blue contain vehicles, whereas regions labeled in red contain no vehicles.

The visualizations also indicate the potential as well as the limitation of the proposed SAR module to reduce the inference time. While in case of rural areas (see Figure 7.10) the majority of the tiles contain no vehicle and are filtered out, DLR 3K comprises images with high traffic volume across the entire scene, so that the search area is only marginally reduced (see Figure 7.9 bottom row). As the computational costs for the RPN and classification stage and consequently the inference time depend on the number of tiles without vehicles that are filtered out, the speed-up increases with fewer tiles considered for detection. Figure 7.11 shows the relation between speed-up compared to Faster R-CNN without SAR and the number of tiles filtered out exemplarily for DLR 3K on the server setup. For this, the number of tiles considered for detection is fixed for each image and not based on the confidence score about the presence of at least one vehicle. The timings are performed accordingly to Section 5.2.4. Due to the auxiliary computational costs of the SAR module, the removed search area has to comprise at least 10 tiles, which is 40% of the image, to reduce the inference time. However, in case of images that comprise only a few tiles that contain at least one vehicle, the inference time decreases considerably, e.g., more than 125 ms if only 20% of the image regions are considered for detection. Note that the time spent for the auxiliary components and the proposal layer may be further reduced by implementing these layers in CUDA to allow computation on a GPU.

**Figure 7.11:** Speed-up in inference time between Faster R-CNN with and without SAR with respect to the number of tiles filtered out. Due to the additional computational costs of the SAR module, at least 10 tiles, which is 40% of the image, have to be filtered out to reduce the inference time.

## **8 Evaluation**

In this chapter, the methods proposed within the context of this thesis are extensively evaluated following the evaluation protocol given in Section 4.2. First, different combinations of the methods proposed for improving the detection accuracy (see Chapter 6) and the methods proposed for runtime optimization (see Chapter 7) are examined in Section 8.1. Comparison of the combined methods to representative existing work in the literature with respect to detection accuracy and inference time is provided in Section 8.2. To demonstrate the generalization ability of the proposed methods, additional experiments are conducted in qualitative manner on different aerial imagery datasets in Section 8.3. Finally, a summary is given in Section 8.4.

## **8.1 Combined Methods for Improved Detection and Inference Time**

As real-world applications generally require high detection accuracies at realtime or near real-time, the methods proposed for improving the detection accuracy by integrating contextual knowledge and the methods proposed for runtime optimization are combined in this section. For this, all experiments are conducted on the DLR 3K dataset. Note that data augmentation is performed for all trainings by applying vertical and horizontal flipping as well as rotation in steps of 90 degrees.

#### **Combined Integration of Spatial and Semantic Context**

As demonstrated in Section 6.1 and Section 6.2, integrating spatial and semantic context information results in improved detection accuracy. In the following, the effect of combining methods that either integrate spatial or semantic information are examined. To this end, MFD Faster R-CNN and EMT Faster R-CNN are merged into a single network as depicted in Figure 8.1. Instead of employing the outputs of *conv4\_3* and *conv5\_3* from the semantic labeling branch as auxiliary feature maps for the classification stage by extracting the corresponding features for each candidate region via RoI pooling, the CEM introduced in Section 6.1.1 is inserted to create a single high-resolution feature map. At first, the features from *conv5\_3* are up-sampled and combined with the features from *conv4\_3*. Then, the combined features are up-sampled and merged with the output from *conv3\_3*. Thus, the semantically enriched features of deeper layers from the semantic labeling branch are propagated to the resulting feature map. Note that the proposed combination facilitates the use of the resulting feature map for both the RPN and classification stage, so that the candidate region generation is not only implicitly but also explicitly affected by the semantic labeling branch. The resulting network is trained end-to-end using the joint multi-task loss given in eq. (6.2). Weights pre-trained on ImageNet are used for initialization, while all further settings are adopted from Section 6.2.3.

**Figure 8.1:** Schematic illustration of the merged MFD Faster R-CNN and EMT Faster R-CNN. Note that the outputs of *conv4\_3* and *conv5\_3* from the semantic labeling branch are used as inputs for the CEM to create a single high-resolution feature map.

**Table 8.1:** AP (in %) of the combined EMT-MFD Faster R-CNN for various GSDs. Compared to its individual components and the baseline Faster R-CNN, the detection accuracy is improved for all GSDs.

Table 8.1 reports the detection accuracy of the merged MFD Faster R-CNN and EMT Faster R-CNN for various GSDs. Compared to its plain counterparts and Faster R-CNN, the detection accuracy is improved for all GSDs. Note that the AP for MFD Faster R-CNN is higher in comparison with the AP reported in Table 6.1 due to the performed data augmentation. While the gain in AP is minor for a GSD of 13 cm, the improvement becomes more notable with higher GSDs. For a GSD of 26 cm, MFD Faster R-CNN and EMT Faster R-CNN are outperformed by 1.9% and 1.5% in AP, respectively. This shows that inducing scene knowledge via semantic labeling yields more distinctive feature representations in deep layers as expected. Furthermore, it indicates that combining features from different layers via the CEM and the use of the resulting feature map for both stages are beneficial towards the simple integration of features from deep layers for classification via RoI pooling as carried out in EMT Faster R-CNN.

### **Integration of Semantic Context and Lightweight Feature Extraction**

The effect of combining the integration of semantic context and the lightweight feature extraction is examined by replacing the default base network of EMT Faster R-CNN, i.e., VGG16, with the computation-efficient ZynqNet architecture. The outputs of the 7ᵗʰ and 9ᵗʰ Fire module are considered as auxiliary feature maps for the classification stage to explicitly integrate features from the semantic labeling branch. Additional RoI pooling layers are applied to extract the corresponding features for each candidate region, which are fused with the respective output of the RoI pooling layer of the detection branch via element-wise addition as described in Section 6.2.4. VGG0.5-Head is employed as classification head, as it exhibits the best tradeoff between detection accuracy and inference time (see Table 7.11). Joint multi-task loss (see eq. (6.2)) and the settings specified in Section 7.1.5 are adopted for the training.

**Table 8.2:** Comparison of the detection accuracy and inference time for EMT Faster R-CNN with different base networks.

The AP and inference times for EMT Faster R-CNN with ZynqNet=1.00 as base network are given in Table 8.2. In comparison to EMT Faster R-CNN with VGG16, the inference time is reduced by 46% and 68% for the server and desktop setup, respectively, while the detection accuracy only marginally decreases. This demonstrates that even lightweight architectures are suited as base network for EMT Faster R-CNN and thus, allow improved detection accuracy by integration of semantic context with clearly decreased inference time compared to the original Faster R-CNN. Removing filters from the base network following the pruning strategy introduced in Section 7.1.3 further boosts the inference speed. However, the detection accuracy clearly drops with fewer filters per convolutional layer. Table 8.3 illustrates the applicability of EMT Faster R-CNN with ZynqNet=1.00 for higher GSDs. Though the gap in AP compared to its counterpart using VGG16 as base network increases with higher GSD, the AP is still high for a GSD of 26 cm.

**Table 8.3:** AP (in %) for EMT Faster R-CNN using VGG16 and ZynqNet as base network with respect to various GSDs.

### **Integration of Semantic Context and Search Area Reduction**

Figure 8.2 depicts the integration of the SAR module proposed in Section 7.2 into EMT Faster R-CNN. To minimize the auxiliary computational costs for the SAR module, the convolutional layers are shared between the semantic labeling branch and the SAR module. Stage-wise training is performed according to Section 7.2.2, as Faster R-CNN requires at least one GT object per training image, whereas the SAR classifier requires images with and without GT objects. In the initial stage, EMT Faster R-CNN is trained using the settings specified in Section 6.2.3, while the SAR classifier is trained in the second stage as described in Section 7.2.2. Note that all convolutional layers are kept fixed during the second training stage. For inference, the further components of the SAR module, i.e., the sub-division-SAR layer and the sub-division-RPN layer, and the modified proposal layer are added to the framework. All settings are adopted from Section 7.2.2.

As shown in Table 8.4, the times spent for region proposal generation and the classification stage as well as the overall inference time are clearly reduced by restricting the search area. Note that the detection accuracy marginally drops by 0.1% in AP, as the number of false negatives increases due to incorrectly classified image areas. Compared to EMT Faster R-CNN with ZynqNet as base network, the detection accuracy is on par, while the gain in inference time is smaller because of the high traffic volume in the DLR 3K dataset.

### **Integration of Spatial Context and Lightweight Feature Extraction**

To combine the integration of spatial context and the lightweight feature extraction, ZynqNet is applied as base network for MFD Faster R-CNN. The integration of spatial context from deeper layers is conducted in a stage-wise manner by applying the CEM introduced in Section 6.1.1. The output from the 9ᵗʰ Fire module is up-sampled and combined with the output from the 7ᵗʰ Fire module. The combined features are up-sampled and combined with the output from the 5ᵗʰ Fire module. The resulting feature map enriched with features that comprise more contextual information is then used as input for the RPN and classification stage. Due to its good trade-off between detection accuracy and inference time in case of Faster R-CNN with ZynqNet as base network, VGG0.5-Head is employed as classification head. For training, the staged fine-tuning scheme proposed in Section 6.1.2 is adopted.

The AP and inference times for MFD Faster R-CNN with ZynqNet=1.00 as base network are given in Table 8.5. The inference time is significantly reduced on both setups by replacing the default base network with ZynqNet=1.00, while the decrease in AP is small. Applying the pruning strategy introduced in Section 7.1.3 to remove redundant filters yields a further acceleration as expected. The drop in AP with fewer filters per convolutional layer is smaller compared to EMT Faster R-CNN with ZynqNet (see Table 8.2).

**Table 8.5:** Comparison of the detection accuracy and inference time for MFD Faster R-CNN with different base networks.

**Table 8.6:** AP (in %) for MFD Faster R-CNN using VGG16 and ZynqNet as base network with respect to various GSDs.

The applicability of MFD Faster R-CNN with ZynqNet=1.00 for higher GSDs is demonstrated in Table 8.6. While the detection accuracy drops by 0.7% in AP compared to MFD Faster R-CNN with VGG16 for a GSD of 19.5 cm, the drop in AP is only 0.4% for a GSD of 26 cm.

### **Integration of Spatial Context and Search Area Reduction**

**Figure 8.3:** Schematic illustration of the MFD Faster R-CNN with SAR. Note that the combined feature map is divided into tiles and region proposals are only generated for tiles that are likely to contain at least one object based on the SAR classification.

The SAR module is added to the MFD Faster R-CNN as illustrated in Figure 8.3. The model is trained in a stage-wise manner. MFD Faster R-CNN is initially trained according to Section 6.1.2. In the second stage, the SAR classifier is trained as specified in Section 7.2.2, whereby all convolutional layers are kept fixed. For deployment, the further components of the SAR module and the modified proposal layer are added to the framework and the settings stated in Section 7.2.2 are adopted.

**Table 8.7:** Comparison of the inference time for MFD Faster R-CNN with and without SAR. The overall inference time and in particular the inference time for the RPN and the classification stage are reduced by applying the SAR module.

The impact of the SAR module on the inference time is reported in Table 8.7. The time spent for the RPN and classification stage on the server setup are reduced by 32% and 76%, respectively, yielding an overall decrease of 21%, while the detection accuracy slightly drops by 0.1% in AP, which is on par with the detection accuracy achieved for MFD Faster R-CNN with a lightweight base network (see Table 8.5).

### **Lightweight Feature Extraction and Search Area Reduction**

The usage of lightweight architectures as base network as well as the restriction of the search area exhibit an improved inference time. To further speed up the inference time, the SAR module is inserted into Faster R-CNN and the default base network is replaced by ZynqNet=1.00. To this end, the output from the 5ᵗʰ Fire module is considered as feature map for the RPN and classification stage. The auxiliary Fire modules, i.e., the 6ᵗʰ through the 9ᵗʰ Fire module, followed by max pooling, a sequence of fully connected layers and a softmax layer are employed as SAR classifier. For this, the sequence of fully connected layers is adopted according to Section 7.2.1. VGG0.5-Head is employed as classification head, which exhibits the best trade-off between inference time and AP. The model is trained in two stages as described in Section 7.2.2. The detection part is first trained with the settings specified in Section 7.1.5, while the SAR classifier is trained as described in Section 7.2.2. The further components of the SAR module and the modified proposal layer are inserted for deployment as shown in Figure 7.7. Furthermore, the settings for deployment proposed in Section 7.2.2 are applied.

Table 8.8 shows the overall inference time and the time spent for each component of Faster R-CNN with SAR and lightweight feature extraction. Inserting the SAR module into Faster R-CNN with ZynqNet as base network achieves overall the best inference time. The usage of the lightweight base network results in clearly reduced time spent for feature extraction. Furthermore, the time spent for classification decreases due to the more lightweight classification head. Inserting the SAR module into Faster R-CNN with ZynqNet as base network results in clearly less time spent for generating region proposals and classification compared to its counterpart without SAR, while the detection accuracy only worsens by 0.1% in AP. In comparison to the baseline Faster R-CNN, the overall inference time is speeded up by 53% and 75% on the server and the desktop setup, respectively, as the times spent for each component are clearly reduced, whereas the detection accuracy only drops by 0.2% in AP. Note that all timings are repeated for the models pre-trained on augmented data and thus, the reported inference times may vary slightly compared to their counterparts without augmented data given in previous chapters. The most time-consuming component of Faster R-CNN with SAR and lightweight feature extraction is the RPN. A reason for this is the proposal layer implemented in Python that computes the final coordinates and sorts all proposals by the predicted confidence score on the CPU.

**Table 8.8:** Overall inference time and the time spent for each component of Faster R-CNN with SAR using ZynqNet as base network compared to the baseline Faster R-CNN and the separate approaches proposed to improve the inference time.

#### **Combination of all proposed Approaches**

Finally, all proposed approaches are merged into a single detector as visualized in Figure 3.1. For this, ZynqNet is used as base network. Note that all layers from the initial convolutional layer through the 9ᵗʰ Fire module are shared between the EMT Faster R-CNN, the MFD Faster R-CNN and the SAR classifier to minimize the computational costs. The output from the 9ᵗʰ Fire module is up-sampled and combined with the output from the 7ᵗʰ Fire module. The combined features are up-sampled and combined with the output from the 5ᵗʰ Fire module. The resulting feature map is then used as input for the RPN and classification stage as aforementioned. The final model is trained in two stages. First, EMT Faster R-CNN and MFD Faster R-CNN are trained jointly using the proposed multi-task loss (see eq. (6.2)) and the settings specified in Section 7.1.5. In the second stage, the SAR classifier is trained as described in Section 7.2.2, while the layers shared with EMT Faster R-CNN and MFD Faster R-CNN are kept fixed. For deployment, the further components of the SAR module are inserted analogous to Figure 7.7.

**Table 8.9:** Detection accuracy and inference times for the final model that combines all proposed approaches.

The detection accuracy and the inference times for the final model are reported in Table 8.9. In comparison with the baseline Faster R-CNN, the detection accuracy is improved by 1.1% in AP, while the inference time is reduced by 52% and 67% on the server and desktop setup, respectively. The improved detection accuracy is mainly due to the reduced number of FPs caused by vehicle-like structures (see Figure 8.4). Compared to other combinations of the proposed approaches, the final model exhibits the best trade-off between detection accuracy and inference time. The inference time is considerably reduced compared to the joint EMT-MFD Faster R-CNN with and without SAR, while the detection accuracy only drops by 0.2% in AP. On the other hand, the detection accuracy is improved by 1.3% in AP compared to Faster R-CNN with SAR and ZynqNet as base network. The additional computational costs caused by the modules proposed to integrate spatial and semantic information result in an only marginally increased inference time on the server setup, whereas the inference time increases by roughly 33% on the desktop setup.

**Figure 8.4:** Qualitative detection results (red boxes) and corresponding GT (green boxes) for Faster R-CNN with VGG16 (left column), EMT-MFD Faster R-CNN + SAR with VGG16 (middle column) and EMT-MFD Faster R-CNN + SAR with ZynqNet (right column) on DLR 3K demonstrate that the proposed approaches are more robust to false alarms due to vehicle-like structures.

## **8.2 Comparison to Related Work**

The combined EMT-MFD Faster R-CNN with SAR is in the following compared to representative existing work in the field of object detection in aerial imagery. As most of these approaches adopt deep learning based detection frameworks for the task of vehicle detection in aerial imagery, recently proposed deep learning based detection frameworks are further considered for the comparison. The detection performance for EMT-MFD Faster R-CNN with SAR and the detection methods from literature are given in Table 8.10. The considered vehicle detection methods either adapt the size of the employed feature map [Car17, Aca18, Din18], exploit multiple feature maps [Guo18, Tay18], combine features from different layers [Den17, Din18, Yan18] or make use of a top-down architecture [Aca18, Guo18, Tay18] to account for the characteristics of aerial imagery. Note that the detection methods proposed for vehicle detection in aerial imagery are adopted unmodified. All the additionally used deep learning based detection frameworks have a top-down architecture and exploit multiple feature maps except for Faster R-CNN with OHEM. For each detection framework, the shallowest feature map has been selected in such a way that its dimensions are 1/4 of the input image in order to account for small-sized vehicles. Furthermore, the anchor box scales are adopted according to Table 5.6. To ensure a fair comparison, each model is trained on the identical training data with the same data augmentation settings.

Amongst the vehicle detection methods, the best AP on DLR 3K is achieved for DYOLO and Adapted RetinaNet, which both comprise a top-down architecture similar to MFD Faster R-CNN in order to obtain feature maps enriched with more context information. Shallow YOLOv2, AVPN Faster R-CNN and DFL-CNN exhibit considerably worse AP compared to the other vehicle detection methods. Though both Shallow YOLOv2 and DFL-CNN specifically adapt the resolution of the employed feature map, the resulting resolution, i.e., 1/16 of the input image, which is the same for AVPN Faster R-CNN, is not sufficient to accurately detect small-sized vehicles as in case of DLR 3K. Overall, none of the examined vehicle detection methods accomplishes an AP that is on par with the AP achieved for EMT-MFD Faster R-CNN with SAR. One reason for this is the coarser feature map resolutions applied throughout the different methods, i.e., 1/8 or 1/16 of the input image, which may result in poorly located as well as duplicate detections (see Section 5.2.1). However, it is to mention that some of these approaches are designed for object detection in aerial imagery that comprise larger object instances such as NWPU.

**Table 8.10:** Average precision and inference time for EMT-MFD Faster R-CNN + SAR with VGG16 and ZynNet, respectively, compared to representative existing work on the DLR 3K dataset.

¹ Using the Darknet framework (https://github.com/pjreddie/darknet)

² Using the Caffe framework

³ Using the MMDetection [Che19b] toolbox based on PyTorch (https://github.com/open-mmlab/ mmdetection)

Faster R-CNN with OHEM outperforms the baseline Faster R-CNN by 0.2% in AP (see Table 8.9), which demonstrates the advantage of hard negative mining to learn more robust feature representations. However, the detection accuracy is generally worse compared to the adapted deep learning based detection frameworks that make use of a top-down architecture. Hence, enhancing the context information by combining features of shallow and deep layers possesses a larger impact on the detection accuracy. FPN, which extends Faster R-CNN by a top-down path that combines features from different layers, outperforms the baseline Faster R-CNN by 0.4% in AP. Inserting different variants of deformable convolutions (FPN - DCNv1 and FPN - DCNv2) results in no gain in AP. While Libra R-CNN that aims at reducing the imbalance at sample, feature, and objective level only slightly improves the detection accuracy of the FPN, applying a cascaded training scheme (Cascade R-CNN) in order to increase the localization accuracy outperforms the FPN by 0.3% in AP. The best AP amongst the recently proposed deep learning based detection frameworks is achieved for RefineDet and YOLOv3, which is almost on par with the proposed detection methods. In comparison with the examined vehicle detection methods, the recently proposed deep learning frameworks generally exhibit higher AP values. This shows that the conducted adaptations proposed in Section 5.2 are transferable to more recent detection frameworks and are essential to achieve state-of-the-art detection accuracies in case of tiny objects.

The best inference time is achieved for Adapted RetinaNet followed by YOLOv3 and FPN. Both RetinaNet and YOLOv3 perform detection in a single stage, which is in general less computationally expensive than two-stage approaches. A reason for the high number of FPS in case of FPN, although FPN is an extended version of Faster R-CNN that comprises more computational operations, is the more efficient implementation of the proposal generation step. While the Caffe implementation performs the proposal generation on the CPU, the MMDetection [Che19b] toolbox implementation runs completely on the GPU. Hence, implementing the components proposed in this thesis with MMDetection in future work is an opportunity to further speed up the inference time of EMT-MFD Faster R-CNN with SAR. AVPN Faster R-CNN achieves the best inference time amongst the methods implemented in Caffe, which is mainly due to the coarse feature map resolution and consequently, the reduced number of region proposals that have to be processed. However, employing such a coarse feature map resolution is not practicable as discussed above. The best inference time for methods implemented in Caffe, which exhibit a fine feature map resolution, is achieved for the proposed detection method. Even single-stage approaches are outperformed, which demonstrates the benefits of the proposed components.

Comparison of EMT-MFD Faster R-CNN with SAR to existing work from literature for various GSDs on DLR 3K is given in Table 8.11. For this, Adapted RetinaNet, which showed the best AP for a GSD of 13 cm amongst the vehicle detection methods, and the adapted detection frameworks with the highest AP are considered. Note that the anchor box scales are adopted according to Table 5.6. EMT-MFD Faster R-CNN with SAR using VGG16 as base network exhibits the best detection accuracy for all GSDs followed by RefineDet and YOLOv3. Using ZynqNet as base network yields slightly worse AP values for higher GSDs, though ZynqNet comprises considerably fewer parameters compared to the other employed base networks. Adapted RetinaNet exhibits the largest decrease in AP with higher GSDs, which verifies the importance of fine feature map resolutions especially in case of tiny objects.

**Table 8.11:** AP (in %) for EMT-MFD Faster R-CNN + SAR with VGG16 and ZynqNet, respectively, compared to representative existing work on the DLR 3K dataset for various GSDs.

Qualitative detection examples (red boxes) and corresponding GT (green boxes) on DLR 3K with a GSD of 26 cm are visualized in Figure 8.5. While all detection methods show a good localization accuracy, applying the proposed detection method results in fewer false alarms caused by vehicle-like structures on buildings. However, for all methods, remaining false alarms are mainly due to objects located on asphalted areas with vehicle-like shapes.

To demonstrate the transferability of the detection methods proposed in this thesis, MFD Faster R-CNN with SAR is compared to existing work from literature on VEDAI LCI and VEDAI SCI (see Table 8.12). Note that exploitation of semantic information is not conducted because of the missing semantic labeling annotations. Each model is trained with the same data augmentation settings, i.e., vertical and horizontal flipping as well as rotation in steps of 90 degrees. The proposed MFD Faster R-CNN with SAR and VGG16 as base network achieves the best AP on both versions of the dataset. Using ZynqNet as base network exhibits the same detection accuracy on VEDAI LCI, while the detection accuracy slightly drops in case of the higher GSD as observed for DLR 3K as well. Amongst the adapted detection frameworks, RefineDet shows overall the best detection accuracy. Similar to DLR 3K, Adapted RetinaNet exhibits the poorest detection accuracies due to the coarse feature map resolution. Qualitative detection examples (red boxes) and corresponding GT (green boxes) on VEDAI SCI are visualized in Figure 8.6.

**Table 8.12:** AP (in %) for MFD Faster R-CNN + SAR with VGG16 and ZynqNet, respectively, compared to representative existing work on the VEDAI dataset.

**Figure 8.5:** Qualitative detection results (red boxes) and corresponding GT (green boxes) for EMT-MFD Faster R-CNN + SAR with VGG16 and ZynqNet, respectively, and representative existing work on DLR 3K with a GSD of 26 cm.

**Figure 8.6:** Qualitative detection results (red boxes) and corresponding GT (green boxes) for EMT-MFD Faster R-CNN + SAR with VGG16 and ZynqNet, respectively, and representative existing work on VEDAI SCI.

## **8.3 Generalization to Unseen Aerial Imagery**

In the following, the generalization ability of the proposed detection method is demonstrated by auxiliary experiments on three recently published aerial imagery datasets, i.e., ITCVD, DOTA and xView (see Section 4.1). Note that models pre-trained on DLR 3K are employed for all experiments.

#### **ITCVD**

Amongst the three datasets mentioned above, ITCVD is most similar to DLR 3K in terms of image quality and content. Both datasets are acquired over Western European cities and thus, mainly comprise urban and residential areas with comparable structures and objects. Figure 8.7 shows qualitative detection results (red boxes) and corresponding GT (green boxes) for EMT-MFD Faster R-CNN with VGG16 and EMT-MFD Faster R-CNN with ZynqNet on images from the ITCVD test set, whereby images taken in oblique view with a tilt angle of 45 degrees are not considered. During deployment, the test images are down-scaled by a factor of 1.3, so that the GSD is similar to DLR 3K. The proposed methods achieve a good classification and localization accuracy even in case of a weak contrast between vehicle and background. Note that even multiple vehicles are correctly detected, which are missed during the annotation process especially in backyards and entrance areas. Hence, the proposed methods are well transferable to unseen data that comprise similar characteristics compared to the employed training data, i.e., the DLR 3K dataset. As depicted in Figure 8.8, EMT-MFD Faster R-CNN with VGG16 and ZynqNet, respectively, exhibit considerably fewer false alarms caused by vehiclelike structures on buildings compared to Faster R-CNN with VGG16 taken as baseline. Though ITCVD comprises strong parallax effects due to the low acquisition altitude, the proposed methods are robust to false alarms caused by the accompanying disturbing structures that are hardly or non-existent in the training data. However, remaining false positive detections are caused by vehicle-like structures on asphalted areas, whereas missed detections are mainly due to partial occlusion, e.g., by trees, or due to vehicles located in shadowed areas (see Figure 8.9), which is similar to observations on DLR 3K.

**Figure 8.7:** Qualitative detection results (red boxes) and corresponding GT (green boxes) for EMT-MFD Faster R-CNN with VGG16 (top row) and with ZynqNet (bottom row) on the ITCVD dataset show the good detection accuracy. Note that even multiple vehicles with missing annotations are correctly detected.

**Figure 8.8:** Qualitative detection results (red boxes) and corresponding GT (green boxes) for Faster R-CNN with VGG16 (left column), EMT-MFD Faster R-CNN with VGG16 (middle column) and EMT-MFD Faster R-CNN with ZynqNet (right column) on the ITCVD dataset demonstrate that the proposed approaches are more robust to false alarms due to vehicle-like structures in unseen data, while vehicles are correctly detected.

**Figure 8.9:** Qualitative examples of missed detections and false alarms for EMT-MFD Faster R-CNN with VGG16 (top row) and ZynqNet (bottom row) on the ITCVD dataset. Missed detections are mainly due to objects that are partially occluded, e.g., by trees, or that are located in shadowed areas, while false alarms are mostly caused by structures located on asphalted areas.

#### **DOTA**

In contrast to DLR 3K, the DOTA dataset comprises images from multiple sensor platforms and consequently, exhibits varying image qualities, differing scenarios and a larger variety of object appearances. Figure 8.10 depicts qualitative detection results (red boxes) and corresponding GT (green boxes) for EMT-MFD Faster R-CNN with VGG16 and with ZynqNet on images from the DOTA validation set. For this, only GT annotations for vehicle categories, i.e., small vehicles and large vehicles, are visualized, whereas further categories such as *baseball diamond*, *harbor*, *bridge*, etc. are not considered. All images are scaled during deployment to exhibit a similar GSD as in case of DLR 3K. Both EMT-MFD Faster R-CNN with VGG16 and ZynqNet exhibit a good localization and classification accuracy, which indicates the good transferability in case of differing scenarios and poor image quality. Furthermore, issues regarding the provided GT, i.e., poorly aligned bounding boxes and missing annotations, are illustrated, which impedes the validity of a quantitative analysis. Compared to Faster R-CNN with VGG16, applying the proposed methods results in clearly fewer false alarms caused by vehicle-like structures on buildings (see Figure 8.11), which confirms observations on ITCVD.

**Figure 8.10:** Qualitative detection results (red boxes) and corresponding GT (green boxes) for EMT-MFD Faster R-CNN with VGG16 (top row) and with ZynqNet (bottom row) on the DOTA dataset indicate the good detection accuracy in unseen images from different sensors. Note that even vehicles with missing annotations are detected.

**Figure 8.11:** Qualitative detection results (red boxes) and corresponding GT (green boxes) for Faster R-CNN with VGG16 (left column), EMT-MFD Faster R-CNN with VGG16 (middle column) and EMT-MFD Faster R-CNN with ZynqNet (right column) on the DOTA dataset demonstrate that the proposed approaches are more robust to false alarms due to vehicle-like structures in unseen data.

Figure 8.12 shows the main reasons for remaining false alarms and missed detections. While, similar to results on the DLR 3K dataset, missed detections are mainly due to objects that are partially occluded, e.g., by trees, false alarms are mostly caused by shadows or structures located on asphalted areas. Additional false alarms stem from split detections caused by vehicle types that are not represented in the training data.

**Figure 8.12:** Qualitative examples of missed detections and false alarms for EMT-MFD Faster R-CNN with VGG16 (top row) and ZynqNet (bottom row) on the DOTA dataset. Missed detections are mainly due to objects that are partially occluded, e.g., by trees, while false alarms are mostly caused by shadows or structures located on asphalted areas. Furthermore, false alarms are caused by split detections due to vehicle types that are not represented in the training data.

#### **xView**

The xView dataset comprises images acquired over different continents and thus, exhibits a larger variety of scenarios and objects compared to DLR 3K. Qualitative detection results (red boxes) and corresponding GT (green boxes) for EMT-MFD Faster R-CNN with VGG16 and ZynqNet on the xView dataset are depicted in Figure 8.13. For this, only GT annotations for categories belonging to the meta class *passenger vehicle* are visualized. In contrast to the experiments performed on ITCVD and DOTA, models pre-trained on DLR 3K down-scaled by a factor of 2 are employed due to the low GSD of approximately 30 cm. Both EMT-MFD Faster R-CNN with VGG16 and with ZynqNet exhibit almost no false alarms even in scenes with complex backgrounds, while detections are accurately aligned around occurring vehicles. However, multiple vehicles are not detected because of the relatively large differences to the training data, in particular the poorer image quality and larger variety of occurring objects. This indicates that the generalization ability gets worse in case of higher GSDs and consequently smaller object dimensions.

**Figure 8.13:** Qualitative detection results (red boxes) and corresponding GT (green boxes) for EMT-MFD Faster R-CNN with VGG16 (left column) and with ZynqNet (right column) on the xView dataset. Despite the low spatial resolution, the proposed approaches are robust to false alarms due to vehicle-like structures in unseen data with differing scenarios and backgrounds. However, multiple missed detections occur due to the relatively large differences to the training data.

## **8.4 Summary**

Finally, it is worth discussing the strengths and weaknesses of the proposed components and the examined combinations with respect to real-world applications. The CEM introduced to enhance the spatial context information of the employed features is straightforward to integrate into deep learning frameworks, as essentially the feature extraction is altered. Its benefits to the feature representation are demonstrated in the performed experiments by means of clearly reduced false alarms caused by vehicle-like structures in unlikely areas, which yields an improved detection accuracy. The auxiliary computational costs on the other hand only slightly increase the inference time, which may be tolerable for most applications in combination with techniques to optimize the inference time. Implementations similar to the CEM are a major part of the most recent deep learning based detectors that achieve stateof-the-art results in various domains. This further emphasizes the importance of such a component. Inducing scene knowledge via semantic labeling improves the feature representation as well. The detection accuracy is clearly improved, as the number of false alarms caused by vehicle-like structures in unlikely areas is reduced. However, in contrast to the CEM, the training requires semantic labeling annotations, whose generation is time-consuming. Thus, semantic labeling annotations are often not available and the training is restricted to a few aerial imagery datasets. Consequently, the applicability of the semantic labeling based approach may be limited to images similar to these datasets. Since architectures for semantic labeling in aerial imagery are typically developed for low GSDs, novel architectures have to be explored to better account for extremely fine structures as in case of GSDs above 20 cm. Otherwise, the potential of the semantic content information may be not fully exploited. Using computation-efficient CNN architectures considerably reduced the time spent for feature extraction without large drops in detection accuracy, which is essential for applications that have to run in real-time or near real-time. As training deep learning based detection frameworks with such CNN architectures does not depend on specific datasets, its applicability is not restricted. However, the drop in detection accuracy compared to heavyweight CNN architectures increases with higher GSDs and thus, more complex scenarios due to the smaller object dimensions. As one reason for this is the clearly reduced number of parameters used for feature representation, the computation-efficient CNN architectures could be modified to address this issue. The proposed module to restrict the search area to areas of interest results in a large speed-up of the region proposal generation and the classification stage, while the detection accuracy remains almost unaffected. Prerequisite for this speed-up is that only a small fraction of an aerial image is occupied by relevant objects and thus, most image regions are not considered for detection. Therefore, the proposed module is less appropriate for aerial imagery recorded over urban areas with dense traffic volumes. Since a classifier is trained on the respective data to identify areas that are unlikely to contain a vehicle, the generalization ability is further affected by the quality of the classifier, which may limit its applicability to unseen data. A data independent alternative is the integration of referenced road maps to identify areas of interest, whereby road maps are not always provided and vehicles offside roads are missed.

While each component already outperforms the baseline Faster R-CNN either in terms of detection accuracy or inference time, combining the proposed components further boosts the detection performance. Combining both alternative approaches to enhance the feature representation shows an additional gain in detection accuracy, especially in case of more complex scenarios such as higher GSDs. Hence, this combination is particularly of interest for applications relying on very accurate detection. As the computation-efficient CNN architecture and the SAR module reduce the computational costs for different stages of the detection pipeline, their combination further speeds up the overall inference time. Thus, this combination is better suited than the individual approaches for applications with harsher time constraints. All combinations of components to enhance the detection accuracy and components to decrease the inference time yield an improved trade-off between inference time and detection accuracy compared to the individual components, so that these combinations are good alternatives to fulfill time and accuracy constraints. The best trade-off between inference time and detection accuracy is achieved for integrating all components into the detection pipeline, whereby aforemtioned limitations may restrict its applicability. Nevertheless, the stand-alone character of the proposed components allows the usage of themselves or in different combinations in order to meet the specific requirements of an application.

In summary, the proposed detection pipeline comprised of the components to enhance the detection accuracy and inference time outperforms representative existing work from literature on different aerial imagery. Furthermore, a good generalization ability is demonstrated on unseen data with differing scenarios, which is essential for most real-world applications. Compared to the baseline Faster R-CNN especially the number of false alarms is considerably reduced due to the more robust feature representation.

## **9 Conclusions and Outlook**

## **9.1 Conclusions**

In this thesis, a novel deep learning based detection pipeline is proposed for the task of vehicle detection in aerial imagery. For this, Faster R-CNN is systematically adapted with respect to the specific characteristics of aerial imagery. Increasing the resolution of the feature map employed for region proposal generation and classification clearly improves the detection accuracy, as localization issues in case of small-sized objects are solved. Despite the improved detection accuracy, the performed adaptations yield several shortcomings, i.e., low semantic and spatial content of the employed features and a poor inference time, which impede the usage in real-world applications.

Two novel approaches are proposed to overcome the lack of semantic and spatial content and thus, reduce the number of false alarms. The first approach enhances the spatial context information by combining features from different layers to account for fine and coarse structures, while maintaining a high feature map resolution. For this purpose, Faster R-CNN is extended by the proposed CEM, which utilizes deconvolutional layers to up-sample features of deep layers. The latter approach leverages semantic labeling to increase the semantic context information, whereby two variants to integrate semantic labeling into the detection framework are realized. Inducing scene knowledge by explicitly merging the semantic labeling network into the detection framework via shared feature representations outperforms the alternative variant that exploits the semantic labeling results to filter out unlikely predictions. The proposed approaches exhibit clearly improved detection results, in particular for high GSDs and consequently smaller object dimensions. The reason for the improved detection results is the reduced number of false alarms caused by vehicle-like structures located on regions that are unlikely to contain vehicles, e.g., buildings.

In order to reduce the computational effort and consequently, the inference time, two different strategies are pursued in this thesis. The first strategy aims at optimizing the time spent for feature extraction by replacing the default CNN architecture with a lightweight CNN architecture. In combination with further techniques for runtime optimization, i.e., filter pruning, merged batch normalization and exploitation of computation-efficient classification heads, the inference time is considerably reduced, while the detection accuracy remains almost unaffected. The second strategy restricts the search area to areas of interest by identifying and removing areas that are unlikely to contain at least one vehicle. For this, a novel module to classify image areas is explicitly integrated into the detection framework by sharing the convolutional features. As vehicles generally cover only a small fraction of aerial imagery, the computational efforts for the region proposal stage and the classification stage and consequently the inference time are clearly reduced.

To ensure high detection accuracies at real-time or near real-time, the proposed approaches and strategies are combined into a single detection pipeline. The standard Faster R-CNN detector taken as baseline is significantly improved in terms of detection accuracy and inference time. Furthermore, the proposed method outperforms representative existing work from literature on different aerial imagery datasets. Finally, the generalization ability of the proposed method is demonstrated by auxiliary experiments on unseen data with differing scenarios.

## **9.2 Outlook**

Although the proposed method exhibits good results regarding the detection accuracy of vehicles in aerial imagery, further enhancements and extensions are often necessary to meet the requirements of real-world applications.

Though the components proposed to reduce the computational effort result in a large speed-up compared to the baseline Faster R-CNN, the inference time of the entire detection pipeline may not be sufficient for some applications. The main bottleneck is an inefficient proposal generation, which can be addressed by transferring the corresponding processing steps to the GPU, as implemented in novel frameworks like the MMDetection toolbox based on PyTorch [Che19b]. Integrating the components proposed in this thesis into single-stage detection frameworks offers an alternative to overcome issues with regard to the proposal generation. A promising way to decrease the overall inference time is making use of the NVIDIA TensorRT¹ library, which facilitates high-performance inference of different deep learning frameworks. By combining layers and optimizing kernel selection for improved latency, throughput, power efficiency and memory consumption, TensorRT optimizes the inference time of a given pre-trained network. Moreover, TensorRT offers out-of-the-box INT8 quantization and FP16 precision implementations of common layers as further options to accelerate the inference time.

So far, the proposed detection pipeline is limited to a single vehicle class in aerial imagery recorded in top-down view due to the low availability of annotated training data. However, distinguishing between vehicle classes, e.g., car and truck, is highly relevant for applications such as traffic monitoring and management, while an accurate detection in images recorded in oblique view is often prerequisite for applications like disaster relief and search and rescue tasks. As deep learning is largely data-driven, the trend of getting bigger and more extensive datasets, emerging in different computer vision domains including object detection in aerial imagery, facilitates the learning of more complex tasks. The VisDrone dataset [Zhu18] for instance comprises more than 2.6 million annotations for different object categories, e.g., pedestrian, car, truck, etc., in images recorded by UAVs with different perspectives. Besides the new extension options, novel challenges arise that have to be addressed. Especially imbalanced data that may yield biased rules in favor of the majority class and the large variety of object scales ranging from a few to hundreds of pixels complicate the detection task. Furthermore, recent developments in the field of object detection in aerial imagery focus on the transition from axis-aligned bounding boxes to oriented bounding boxes, which

¹ https://developer.nvidia.com/tensorrt

is especially of interest for tracking based applications that depend on orientation information. For this, most components of the base detection pipeline, e.g., proposal generation, RoI pooling, NMS, etc., and the objective function have to be extended regarding the orientation.

In general, training data is limited in its diversity to particular areas and recording conditions, which impairs the generality and transferability of learned models. Though the proposed detection pipeline exhibits good detection results on unseen data with differing scenarios, limitations of its transferability become obviously recognizable especially in case of high GSDs, poor image quality and large variety of occurring objects. To overcome challenges of cross-domain differences, domain adaptation aims at transferring knowledge learned by a particular network on a source domain to a new related target domain. Common techniques attempt to match the distributions of the source and target datasets by minimizing some divergence criterion or make use of generative adversarial networks (GANs) to generate synthetic target data which are somehow related to the source domain. Few-shot learning – the ability to learn from only few labeled samples – is another promising research direction to address these issues, which may allow re-training on target data on the flight.

To improve the detection accuracy, exploiting temporal context across consecutive frames has recently drawn increasing attention in the computer vision community. Besides established approaches to integrate temporal context like recurrent neural networks (RNNs), applying 3 dimensional convolutions is a popular strategy to learn discriminative features along both spatial and temporal dimensions. Since the data recorded for most applications based on aerial imagery generally comprises image sequences, such approaches are promising to improve the detection accuracy in aerial imagery, in particular in case of tiny or partially occluded objects. While data-driven object detection in aerial imagery has been limited to single images in the past, recent datasets such as the VisDrone dataset and the UAVDT dataset [Du18] providing annotations for image sequences allow the usage of multiple frames.

# **Bibliography**

[Du18] DU, Dawei; QI, Yuankai; YU, Hongyang; YANG, Yifan; DUAN, Kaiwen; LI, Guorong; ZHANG, Weigang; HUANG, Qingming and TIAN, Qi: "The unmanned aerial vehicle benchmark: Object detection and tracking". In: *Proceedings of the European Conference on Computer Vision (ECCV)*. 2018, pp. 370–386. [Duc11] DUCHI, John; HAZAN, Elad and SINGER, Yoram: "Adaptive subgradient methods for online learning and stochastic optimization". In: *Journal of Machine Learning Research* 12.Jul (2011), pp. 2121– 2159. [Dud12] DUDA, Richard O; HART, Peter E and STORK, David G: Pattern classification. John Wiley & Sons, 2012. [Egg17] EGGERT, Christian; BREHM, Stephan; WINSCHEL, Anton; ZECHA, Dan and LIENHART, Rainer: "A closer look: Small object detection in faster R-CNN". In: *Proceedings of the IEEE International Conference on Multimedia and Expo (ICME)*. 2017, pp. 421–426. [Eik09] EIKVIL, Line; AURDAL, Lars and KOREN, Hans: "Classificationbased vehicle detection in high-resolution satellite images". In: *ISPRS Journal of Photogrammetry and Remote Sensing* 64.1 (2009), pp. 65–72. [Erd16] ERDELJ, Milan and NATALIZIO, Enrico: "UAV-assisted disaster management: Applications and open issues". In: *Proceedings of the IEEE International Conference on Computing, Networking and Communications (ICNC)*. 2016. [Eve10] EVERINGHAM, Mark; VAN GOOL, Luc; WILLIAMS, Christopher KI; WINN, John and ZISSERMAN, Andrew: "The pascal visual object classes (voc) challenge". In: *International Journal of Computer Vision* 88.2 (2010), pp. 303–338. [Eye18] EYERMAN, J.; CRISPINO, G.; ZAMARRO, A. and DURSCHER, R.: "Drone Efficacy Study (DES): Evaluating the Impact of Drones for Locating Lost Persons in Search and Rescue Events". In: *DJI and European Emergency Number Association* (2018).

[Kem10] KEMBHAVI, Aniruddha; HARWOOD, David and DAVIS, Larry S: "Vehicle detection using partial least squares". In: *IEEE Transactions on Pattern Analysis and Machine Intelligence* 33.6 (2010), pp. 1250–1265. [Kim03] KIM, ZuWhan and MALIK, Jitendra: "Fast vehicle detection with probabilistic feature grouping and its application to vehicle tracking". In: *IEEE*. 2003, p. 524. [Kin14] KINGMA, Diederik P and BA, Jimmy: "Adam: A method for stochastic optimization". In: *arXiv preprint arXiv:1412.6980* (2014). [Klu07] KLUCKNER, Stefan; PACHER, Georg; GRABNER, Helmut; BISCHOF, Horst and BAUER, Joachim: "A 3D teacher for car detection in aerial images". In: *Proceedings of the IEEE International Conference on Computer Vision (ICCV)*. 2007. [Kog18] KOGA, Yohei; MIYAZAKI, Hiroyuki and SHIBASAKI, Ryosuke: "A CNN-based method of vehicle detection from aerial images using hard example mining". In: *Remote Sensing* 10.1 (2018). [Kon16] KONG, Tao; YAO, Anbang; CHEN, Yurong and SUN, Fuchun: "HyperNet: Towards Accurate Region Proposal Generation and Joint Object Detection". In: *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*. 2016, pp. 845–853. [Kri12] KRIZHEVSKY, Alex; SUTSKEVER, Ilya and HINTON, Geoffrey E: "Imagenet classification with deep convolutional neural networks". In: *Advances in Neural Information Processing Systems (NIPS)*. 2012, pp. 1097–1105. [Kum01] KUMAR, Rakesh; SAWHNEY, Harpreet; SAMARASEKERA, Supun; HSU, Steve; TAO, Hai; GUO, Yanlin; HANNA, Keith; POPE, Arthur; WILDES, Richard; HIRVONEN, David, et al.: "Aerial video surveillance and exploitation". In: *Proceedings of the IEEE* 89.10 (2001), pp. 1518–1539.

## **Publications**

# **List of Figures**

## **List of Tables**

## **Acronyms**

## **Table of Symbols**

## **Calligraphic Symbols**

### **Greek Symbols**

#### **Roman Symbols**

### **Karlsruher Schriftenreihe zur Anthropomatik (ISSN 1863-6489)**

#### Philipp Woock **Umgebungskartenschätzung aus Sidescan-Sonardaten für ein autonomes Unterwasserfahrzeug.** ISBN 978-3-7315-0541-9 **Band 26**

Die Bände sind unter www.ksp.kit.edu als PDF frei verfügbar oder als Druckausgabe bestellbar.

Die Bände sind unter www.ksp.kit.edu als PDF frei verfügbar oder als Druckausgabe bestellbar.

**Band 45** Jürgen Beyerer, Tim Zander (Eds.) **Proceedings of the 2019 Joint Workshop of Fraunhofer IOSB and Institute for Anthropomatics, Vision and Fusion Laboratory.** ISBN 978-3-7315-1028-4 **Band 46** Stefan Becker **Dynamic Switching State Systems for Visual Tracking.** ISBN 978-3-7315-1038-3 **Band 47** Jennifer Sander **Ansätze zur lokalen Bayes'schen Fusion von Informationsbeiträgen heterogener Quellen.** ISBN 978-3-7315-1062-8 **Band 48** Philipp Christoph Sebastian Bier **Umsetzung des datenschutzrechtlichen Auskunftsanspruchs auf Grundlage von Usage-Control und Data-Provenance-Technologien.** ISBN 978-3-7315-1082-6 **Band 49** Thomas Emter **Integrierte Multi-Sensor-Fusion für die simultane Lokalisierung und Kartenerstellung für mobile Robotersysteme.** ISBN 978-3-7315-1074-1 **Band 50** Patrick Dunau **Tracking von Menschen und menschlichen Zuständen.** ISBN 978-3-7315-1086-4 **Band 51** Jürgen Beyerer, Tim Zander (Eds.) **Proceedings of the 2020 Joint Workshop of Fraunhofer IOSB and Institute for Anthropomatics, Vision and Fusion Laboratory.** ISBN 978-3-7315-1091-8 **Band 52** Lars Wilko Sommer **Deep Learning based Vehicle Detection in Aerial Imagery.** ISBN 978-3-7315-1113-7

Lehrstuhl für Interaktive Echtzeitsysteme Karlsruher Institut für Technologie

Fraunhofer-Institut für Optronik, Systemtechnik und Bildauswertung IOSB Karlsruhe

Object detection in aerial imagery is essential for a wide range of applications in the field of civil safety and security. However, low spatial resolution originating from the large distance between sensor and ground, capturing conditions and varying scenarios due to different daytimes and regions impede the detection task. This book proposes a novel deep learning based detection method, focusing on vehicle detection in aerial imagery recorded in top view. The base detection framework is extended by two novel components to improve the detection accuracy by enhancing the contextual and semantical content of the employed feature representation, yielding a reduced number of false detections. To reduce the computational effort and consequently the inference time, a lightweight CNN architecture optimized with regard to vehicle detection in aerial imagery is proposed as base architecture and a novel module restricting the search area to areas of interest is introduced. Extensive evaluation demonstrates state-of-the-art performance and the generalization ability on unseen aerial imagery data.

L. Sommer

Deep Learning based Detection in Aerial Imagery

Band 52

ISSN 1863-6489 ISBN 978-3-7315-1113-7 9<PHONE_NUMBER>37


*Article* **Alzheimer Identification through DNA Methylation and Artificial Intelligence Techniques**

**Gerardo Alfonso Perez 1,\* and Javier Caballero Villarraso 1,2**

**Abstract:** A nonlinear approach to identifying combinations of CpGs DNA methylation data, as biomarkers for Alzheimer (AD) disease, is presented in this paper. It will be shown that the presented algorithm can substantially reduce the amount of CpGs used while generating forecasts that are more accurate than using all the CpGs available. It is assumed that the process, in principle, can be non-linear; hence, a non-linear approach might be more appropriate. The proposed algorithm selects which CpGs to use as input data in a classification problem that tries to distinguish between patients suffering from AD and healthy control individuals. This type of classification problem is suitable for techniques, such as support vector machines. The algorithm was used both at a single dataset level, as well as using multiple datasets. Developing robust algorithms for multi-datasets is challenging, due to the impact that small differences in laboratory procedures have in the obtained data. The approach that was followed in the paper can be expanded to multiple datasets, allowing for a gradual more granular understanding of the underlying process. A 92% successful classification rate was obtained, using the proposed method, which is a higher value than the result obtained using all the CpGs available. This is likely due to the reduction in the dimensionality of the data obtained by the algorithm that, in turn, helps to reduce the risk of reaching a local minima.

**Keywords:** algorithm; identification; Alzheimer


**Author details**

Nerija Banaitiene\* and Audrius Banaitis

*Department of Construction Economics and Property Management, Faculty of Civil Engineering, Vilnius Gediminas Technical University, Vilnius, Lithuania*


**4.1 CSR - environmental, social and governance criteria for investors**

Investors have become more intensively interested in knowing about sustainability, social responsibility or good corporate governance of their possible targeted investee. After investment, they need to obtain some background information on the business entities in which they have invested their capital. Environmental, social, and governance (ESG) criteria are methodological tools representing a set of standards for a company's operations that socially conscious investors use to screen potential investments. ESG reports inform about the analysis results of portfolios from different companies based on their sustainability criteria and social impact.

In recent years, the regulatory environment in the European Union has also required far-reaching obligations to disclose ESG-relevant data and to take them into account in investment decisions particularly in the case of institutional investors such as state pension funds and insurance companies [66, 67], so that financial market regulation – in addition to the self-interest of investors in reducing reputation risks, event risks, regulatory risks, technological and legal risks and saturating demand from private investors – provides additional incentives to include ESG criteria in the investment process [66].

Environmental, social, and governance criteria (ESG criteria) are increasingly in focus of companies as growing number of investments are being made based on such criteria [68–70]. Shares and bonds of companies that do good for the environment, society, and their own employees, suppliers, and customers in as many aspects as possible end up in special sustainability funds and receive more attention [71]. Thus, in the United States, for example, the total US-domiciled assets under management using sustainable strategies increased by 38% from USD 8,700bn (2016) to USD 12,000bn (2018) [69]. Thus, the consideration of ESG criteria and indicators is not only a question of ethics and morals because ESG criteria are considered relevant as filters for internal risks (e.g. event risks of assets) and external risks (e.g. climate change) [72, 73]. In the investment management framework, the pursuit of the highest possible risk/return ratio in capital investment may make it also necessary to minimize risk by taking ESG criteria into account [66]. Thus, ESG investing is not an irrational taste-of-assets decision (investor-taste decisions) but may have a performance-relevant effect so that ESG criteria should be considered as an instrument in rational decision-making [74].

Investment decisions today are largely made analytically in the framework of predefined processes analyzing continuously the same key figures more or less software-e-based resulting in the calculation of the risk/return ratios and the selected stocks and assets with the best ratios and indications [75, 76]. Such an approach quantitative investment results, on the one hand, from that globalization simply means that there are many more assets available as possible investment instruments, and it is no longer possible to evaluate them individually [77]. On the other hand, there is also a trend away from investment managers who base their investment decisions on gut instincts [78] or fundamental and technical analysis (discretionary asset management) and towards an automated, data-model-based investment decision process. So, what is required in general is standardized numerical data [79] such as ESG ratings provided from specialized rating agencies intending to assist in the selection and analysis of individual securities as well as funds and portfolios.

#### *Integrity of the Corporate Social Responsibility and Management of Financial Services… DOI: http://dx.doi.org/10.5772/intechopen.101057*

Therefore, the question arises on how ESG performance is measured [80], and to which extent such data is relevant in the investment process, particularly due to empirical results questioning the use for evaluating ESG risk exposure. Thus, Hübel and Scholz [70], for example, provide significant evidence by testing several different factor models finding that ESG ratings (in this case provided by EIKON) do not provide additional information useful in the management of ESG risks. Furthermore, they found no systematic ESG-related discount or risk premium [70] concluding that "investors can measure the ESG risk exposures of all firms in their portfolios using only stock returns, so that even stocks without qualitative ESG information can be easily considered in the management of ESG risks" [70]. From these results, they conclude that stock prices provide sufficient information also for the evaluation of ESG risk exposure [70]. Moreover, researchers [81, 82] find evidence that ESG ratings from different rating agencies show high positive correlations so that a data collection bias can be excluded as well, so that rating agency methodologies do not explain differences in research results from any form of selection bias.

Other recent studies have based their research on ESG indication effects on risk and performance by collecting primary data mainly from company reports such as [83] developing a carbon factor or using an individual methodology for calculating ESG ratings from company reports [84]. Both studies also provide evidence that ESG indications based on a larger number of indicators do not provide added value information. Nevertheless, selected data from company ESG reporting, particularly in the area of environmental criteria may provide information on ESG risks not included in stock price data [70, 84] recommending a different approach for future research and also risk management practitioners' alternative to the use of agency ESG ratings or only the agencies'sub-rating data.

It is thus not surprising that a recent Deutsche Bank Research study finds that although nine of ten of the world's largest fund managers claim to have a responsible investing mandate, just two-fifths actually use ESG factors in the stock analysis. The study concludes that obviously, investment analysts have trouble translating non-financial information in terms of ESG data into investment decisions [79]. This leads to a follow-up question, how portfolio management practitioners in the area institutional investment use overall ESG performance ratings, or only parts of them or even completely different metrics to assess and manage ESG risks, because it seems to be obvious that agency ESG ratings have their limits in providing additional information on ESG risk exposure not included in market data. Yet, the research results on the integration of ESG information in the investment process of professional investors are rare in contrast to the more extensive research, including performance data and ESG ratings of funds or stocks. Besides the fact, that such research – as discussed above – provides only ambiguous findings on the performance effect of ESG ratings, the question remains unanswered how ESG data are included in the risk management process [71].

#### **4.2 CRS concept cases in accounting and finance services environment**

Digital and blockchain technology offers variety of utilization opportunities in many economic areas, such as accounting, finance, telecommunications, supply chain management, etc.. It requires understanding how new digital tools based on the progressive digital technology function and operate. A blockchain as a digital record of transactions, consists of records labeled as "blocks", which are then linked together in a chain-like formation with a great deal of use. It is a financial domain with many applications. The following **Table 7** summarizes benefits of blockchain implementation into daily management activities (**Table 8**).

#### **Table 7.**

*Source: Elaborated by the authors based on [66].*

#### **Table 8.**

*CSR and blockchain elaborated by authors based on [85–89].*

Supply chain example:

A knowledgeable purchase manager designs an economic plan to eliminate waste and ongoing use of resources. Companies can act dishonestly, continue taking part in harmful activities very often while pretending environmental and social sustainability. Blockchain's decentralized structure ensures that each participant oversees their processes and has access to appropriate data. "Procurement Blockchain" guarantees transparency, and acting on manager's behalf when making purchases, negotiating different terms with suppliers, a procurement deal is based on total ecosystem volume (i.e. purchase data from purchaser and their partners). A blockchain-based database is capable of storing relevant data from all business partners, summarizing total volume of purchases, regardless of who managed the purchase activity. It enables calculating the exact volume discount based on the total purchase, in addition mathematical proof of a correct computation is provided. For instance, 25% reduction of costs of invoice factoring can be achieved as blockchain lowers the risk of selling multiple invoices. With blockchain technology operational data are constantly shared, no need for data-crosschecking – it enables audits to be conducted automatically, price verification process will be eliminated. With this detailed tracking and verification process, linking and sharing can be done synchronously. During the supply chain process, the details of every transaction are recorded and made accessible via a permanent history – once the data is created, it cannot be deleted or altered [86].

The following diagram illustrates how blockchain technology impact at nearly every element of the complex Procure-to-Pay (PTP) process (see the diagram) (**Figure 4**).

Managers can benefit from blockchain technology in the management process, for instance, cost-savings, achieving increased operational efficiency or forming

*Integrity of the Corporate Social Responsibility and Management of Financial Services… DOI: http://dx.doi.org/10.5772/intechopen.101057*

#### **Figure 4.**

*Blockchain Technology for Procure-to-pay process – Accenture source: [87].*

new operational models within management of working capital cycle, specifically in the following areas of supply chain management: procurement, provenance, product authenticity and traceability, digital payments and contracts, logistics and manufacturing.

#### A.**Blockchain and CSR in Working Capital Cycle management Advantages**

	- Smart contracts can help reduce data redundancy across trading partners and eliminate costly mistakes. Can automate the invoicing process and "patch" the expensive procure-to-pay gaps.

"The smart contract: the brain of the blockchain It is a computer protocol designed to automate the execution of the terms of a contract. Once coded and deployed on the blockchain, the smart contract is tamper-proof, autonomous in its execution, verifiable by all parties of the network because it is distributed [91]".

	- detailed tracking and verification process
	- linking and sharing done synchronously
	- eliminating human errors, reducing costs, minimizing time delays associated with performing transactions.