The CPU model of the test computer is i7-10700k, the GPU model is NVIDIA RTX3090 and the memory size is 30 GB; the operating system is Ubuntu18.04, the platform for deep learning uses the TensorFlow, the initial learning rate is set to 0.01 and it is reduced by 5% after each epoch. The number of closest points K is set to 16, the batch processed per iteration is set to 8 and a fixed number of points (approximately 105 ) are sampled as input. The SemanticKITTI and Semantic3D datasets are used as the training and testing sets, respectively.

Table 1 shows the results of the training with different approaches on the semantickitti dataset. The average intersection ratio (mIoU) of all categories is used as a quasi-index, and the parameter column refers to the number of network parameters in the algorithm. It can be seen that the mIoU of all categories obtained by RandLa-Net is significantly better than that of other algorithms, and RangeNet53++ has the best segmentation accuracy for small targets such as traffic signs and bicycles [24], which is because the network parameters of RangeNet53++ [24] are more than 40 times higher than those of RandLa-Net.

**Table 1.** Quantitative results of different approaches on semantickitti.

Table 2 shows the train results of different approaches on Semantic3D, and the mean cross-over-union ratio (mIoU) and overall accuracy (OA) for all classes were used as standard metrics.

RandLa-Net significantly outperforms other algorithms in mIoU and OA. The test accuracy of different methods for eight kinds of targets can be seen in Table 2, and the test accuracy of RandLa-Net algorithm is also better than most algorithms. The point cloud segmentation effect on different datasets is shown in Figure 12.

segmentation effect on different datasets is shown in Figure 12.

**Table 2.** Quantitative results of different approaches on Semantic3D.

standard metrics.

**Figure 12.** The segmentation effect on different datasets: (**a**) frame-by-frame segmentation effect of SemanticKITTI-11; (**b**) frame-by-frame segmentation effect of SemanticKITTI-12; (**c**) panoramic split effect of Semantic3D-1; and (**d**) panoramic split effect of Semantic3D-2. **Figure 12.** The segmentation effect on different datasets: (**a**) frame-by-frame segmentation effect of SemanticKITTI-11; (**b**) frame-by-frame segmentation effect of SemanticKITTI-12; (**c**) panoramic split effect of Semantic3D-1; and (**d**) panoramic split effect of Semantic3D-2.

terrain 17.6 30 53.7 63.6 64.5 72.1 person 0.2 0.9 12.9 31.8 38.3 53.4 bicyclist 0.2 1 13.1 33.6 38.8 53.36 motorcyclist 0 0 0.9 4 4.8 7.2 fence 12.9 16.9 29.9 52.3 58.6 44.5 pole 2.4 6 17.8 36 47.9 51.3 traffic sign 3.7 8.9 24.5 50 55.9 38.6

Table 2 shows the train results of different approaches on Semantic3D, and the mean cross-over-union ratio (mIoU) and overall accuracy (OA) for all classes were used as

**Methods SnapNet ShellNet GACNet KPConv RandLa-Net**  mIoU(%) 59 69.1 70.7 74.6 77.4 OA(%) 88.6 91.8 94 92.9 94.8 man-made terrain 81 86.4 96.4 90.9 94.2 natural terrain 77.2 77.7 92.6 82.8 91.4 high vegetation 79.7 88.5 87.9 84.1 82.9 low vegetation 22.9 60.6 44 47.8 52 buildings 91.1 94.2 83.2 94.5 94.7 hard scape 18.4 37.3 31 40 54.9 scanning artefacts 37.3 43.5 63.5 77.3 70.9 Cars 64.4 77.8 76.2 79.7 76.9

RandLa-Net significantly outperforms other algorithms in mIoU and OA. The test accuracy of different methods for eight kinds of targets can be seen in Table 2, and the test accuracy of RandLa-Net algorithm is also better than most algorithms. The point cloud

Table 3 shows the total time and memory consumption of different methods. It can be seen in Table 3. RandLa-Net has the shortest processing time and the most maximum inference points. Although the number of network parameters of the SPG algorithm is the least, the processing time of point clouds is very long and the overall effect is not as good as RandLa-Net, due to the complex geometric division and hypergraph construction steps. Therefore, RandLa-Net is the most efficient network. Table 3 shows the total time and memory consumption of different methods. It can be seen in Table 3. RandLa-Net has the shortest processing time and the most maximum inference points. Although the number of network parameters of the SPG algorithm is the least, the processing time of point clouds is very long and the overall effect is not as good as RandLa-Net, due to the complex geometric division and hypergraph construction steps. Therefore, RandLa-Net is the most efficient network.

**Table 3.** Efficiency of the semantic segmentation of different methods on sequence 08.

### **5. Comprehensive Test of Automatic Classification and Extraction of Raw Point Cloud Data**

#### *5.1. The Flow of the Algorithm*

The improved RandLa-Net algorithm incorporated with NDT registration can directly process the complete point cloud map, and further obtain the data required by the highdefinition map. The specific algorithm flow is shown in Figure 13.

**Table 3.** Efficiency of the semantic segmentation of different methods on sequence 08.

PointNet 192 0.8 0.49 PointNet++ 9831 0.97 0.98 PointCNN 8142 11 0.05 SPG 43584 0.25 - KPConv 717 14.9 0.54

Net 185 1.24 1.03

the high-definition map. The specific algorithm flow is shown in Figure 13.

**5. Comprehensive Test of Automatic Classification and Extraction of Raw Point Cloud**

The improved RandLa-Net algorithm incorporated with NDT registration can directly process the complete point cloud map, and further obtain the data required by

**(Seconds) Parameters (Millions) Maximum Inference Points**

**(Millions)**

**Total Time**

**Figure 13.** Process of the improved RandLa-Net algorithm. **Figure 13.** Process of the improved RandLa-Net algorithm.

original frame-by-frame data of the 03 point cloud.

## *5.2. Test Data*

RandLa-

*5.1. The Flow of the Algorithm*

**Data**

*5.2. Test Data*  Take the SemanticKITTI Dataset 03 as an example for testing. Figure 14 shows the Take the SemanticKITTI Dataset 03 as an example for testing. Figure 14 shows the original frame-by-frame data of the 03 point cloud. *Electronics* **2022**, *11*, 2795 14 of 19

**Figure 14.** Original dataset. **Figure 14.** Original dataset.

#### *5.3. Point Cloud Mapping 5.3. Point Cloud Mapping*

**Figure 15.** Point cloud mapping.

*5.4. Random Sampling*

will solve this problem.

**Figure 16.** Random sampling.

Firstly, load the rosbag of the original dataset, then use the NDT algorithm to perform the coordinate transformation and registration on the frame-by-frame point cloud data, and build a map globally. The output effect is shown in Figure 15. Firstly, load the rosbag of the original dataset, then use the NDT algorithm to perform the coordinate transformation and registration on the frame-by-frame point cloud data, and build a map globally. The output effect is shown in Figure 15.

In order to reduce the amount of computation, the random sampling of data is required before semantic segmentation, and the sampling effect is shown in Figure 16. We can see that the number of point clouds is significantly reduced and the features become blurred. However, the local feature aggregator of the subsequent RandLa-Net algorithm

**Figure 15.** Point cloud mapping. **Figure 15.** Point cloud mapping. **Figure 15.** Point cloud mapping.

**Figure 14.** Original dataset.

**Figure 14.** Original dataset.

*5.3. Point Cloud Mapping*

*5.3. Point Cloud Mapping*

#### *5.4. Random Sampling 5.4. Random Sampling 5.4. Random Sampling*

In order to reduce the amount of computation, the random sampling of data is required before semantic segmentation, and the sampling effect is shown in Figure 16. We can see that the number of point clouds is significantly reduced and the features become blurred. However, the local feature aggregator of the subsequent RandLa-Net algorithm will solve this problem. In order to reduce the amount of computation, the random sampling of data is required before semantic segmentation, and the sampling effect is shown in Figure 16. We can see that the number of point clouds is significantly reduced and the features become blurred. However, the local feature aggregator of the subsequent RandLa-Net algorithm will solve this problem. In order to reduce the amount of computation, the random sampling of data is required before semantic segmentation, and the sampling effect is shown in Figure 16. We can see that the number of point clouds is significantly reduced and the features become blurred. However, the local feature aggregator of the subsequent RandLa-Net algorithm will solve this problem.

Firstly, load the rosbag of the original dataset, then use the NDT algorithm to perform the coordinate transformation and registration on the frame-by-frame point cloud data,

Firstly, load the rosbag of the original dataset, then use the NDT algorithm to perform the coordinate transformation and registration on the frame-by-frame point cloud data,

and build a map globally. The output effect is shown in Figure 15.

and build a map globally. The output effect is shown in Figure 15.

*Electronics* **2022**, *11*, 2795 14 of 19

**Figure 16.** Random sampling. **Figure 16.** Random sampling. **Figure 16.** Random sampling.

#### *5.5. Point Cloud Semantic Segmentation*

Load the bin format file of the global point cloud image, and use the RandLa-Net algorithm to semantically segment the point cloud image. Figure 17a is the visualization of semantic segmentation on the dataset using our improved RandLa-Net algorithm, and Figure 17b is the visualization of semantic segmentation on the same dataset using the original RandLa-Net algorithm. Table 4 shows the specific effect of the test, which is reflected from the indicator mIoU. We can see that the segmentation effect does not decrease the accuracy due to the change from frame-by-frame segmentation to global segmentation.

Figure 17a is the visualization effect of the global point cloud map classification using RandLa-Net fused with NDT. Figure 17b is the visualization effect of the frame-by-frame point cloud classification using the original RandLa-Net network. Figure 17c is the label information corresponding to each color in the visualization. It can be seen that the algorithm in this paper can be used to classify the point cloud map at one time, and then the required area can be directly extracted. However, since the point cloud data processed by the original RandLa-Net algorithm was not registered and then there are a large number of redundancy points so that the point cloud of each road element cannot be directly extracted for making high-definition maps.

Load the bin format file of the global point cloud image, and use the RandLa-Net algorithm to semantically segment the point cloud image. Figure 17a is the visualization of semantic segmentation on the dataset using our improved RandLa-Net algorithm, and Figure 17b is the visualization of semantic segmentation on the same dataset using the original RandLa-Net algorithm. Table 4 shows the specific effect of the test, which is reflected from the indicator mIoU. We can see that the segmentation effect does not decrease the accuracy due to the change from frame-by-frame segmentation to global

*5.5. Point Cloud Semantic Segmentation*

**Figure 17.** The comparison of point cloud semantic segmentation results: (**a**) segmentation visualization based on the improved RandLa-Net; (**b**) segmentation visualization based on the original RandLa-Net; and (**c**) the label information corresponding to the color. **Figure 17.** The comparison of point cloud semantic segmentation results: (**a**) segmentation visualization based on the improved RandLa-Net; (**b**) segmentation visualization based on the original RandLa-Net; and (**c**) the label information corresponding to the color.

**Table 4.** Test accuracy on semantickitti 03. **Table 4.** Test accuracy on semantickitti 03.

segmentation.

From Table 4, it can be seen that the mIoU of all categories are improved after using the new method. This is because the shape features of each element on the global point cloud map are more complete, which is more conducive to be identified and extracted. However, in the classification of some small targets, the accuracy of the original algorithm is not as high as the frame-by-frame segmentation. This is because the registration of small targets in the registration time is not complete, which leads to some deviation in the subsequent classification.

#### *5.6. Point Cloud Extraction 5.6. Point Cloud Extraction*

Since the file is in bin format, in order to find the point cloud data corresponding to the desired label from the classified point cloud map, the point cloud can be extracted according to the label. Figure 18a is the point extracted on label 15 cloud data, and Figure 18b is the point cloud data extracted on label 9. Since the file is in bin format, in order to find the point cloud data corresponding to the desired label from the classified point cloud map, the point cloud can be extracted according to the label. Figure 18a is the point extracted on label 15 cloud data, and Figure 18b is the point cloud data extracted on label 9.

directly extracted for making high-definition maps.

*Electronics* **2022**, *11*, 2795 16 of 19

motorcyclist 8.9 9.7 fence 47.2 48.3 pole 53.1 53.3 traffic sign 40.4 41.7

Figure 17a is the visualization effect of the global point cloud map classification using RandLa-Net fused with NDT. Figure 17b is the visualization effect of the frame-by-frame point cloud classification using the original RandLa-Net network. Figure 17c is the label information corresponding to each color in the visualization. It can be seen that the algorithm in this paper can be used to classify the point cloud map at one time, and then the required area can be directly extracted. However, since the point cloud data processed by the original RandLa-Net algorithm was not registered and then there are a large number of redundancy points so that the point cloud of each road element cannot be

From Table 4, it can be seen that the mIoU of all categories are improved after using the new method. This is because the shape features of each element on the global point cloud map are more complete, which is more conducive to be identified and extracted. However, in the classification of some small targets, the accuracy of the original algorithm is not as high as the frame-by-frame segmentation. This is because the registration of small targets in the registration time is not complete, which leads to some deviation in the

**Figure 18.** Point cloud extraction: (**a**) vegetation; and (**b**) road. **Figure 18.** Point cloud extraction: (**a**) vegetation; and (**b**) road.

subsequent classification.

Figure 18 is the visual effect of extracting all the points marked vegetation and road in the SemanticKITTI dataset 03. It can also be converted into txt type and data in (x, y, z, label) format can be obtained for further operation. In addition to the two extraction examples of Figure 18, specific extractions can be performed based on 19 labels. The extracted data can be directly used to create high-definition maps after simple sorting. The sorted part of the data is shown in Table 5, where the category corresponding to the number is represented by a label. Figure 18 is the visual effect of extracting all the points marked vegetation and road in the SemanticKITTI dataset 03. It can also be converted into txt type and data in (x, y, z, label) format can be obtained for further operation. In addition to the two extraction examples of Figure 18, specific extractions can be performed based on 19 labels. The extracted data can be directly used to create high-definition maps after simple sorting. The sorted part of the data is shown in Table 5, where the category corresponding to the number is represented by a label.

**Table 5.** The form after data output.

The data are labeled in the following order: 1. car; 2. bicycle; 3. motorcycle; 4. truck; 5. other-vehicle; 6. person; 7. bicyclist; 8. motorcyclist; 9. road; 10. parking; 11. sidewalk; 12. other-ground; 13. building; 14. fence; 15. vegetation; 16. trunk; 17. terrain; 18. pole; and 19. traffic sign.

#### **6. Conclusions**

This paper proposes a fast automatic classification and extraction method for largescale point cloud data. The original point cloud data are globally mapped by NDT registration. In order to reduce the amount of computation, the point cloud map is randomly sampled. The cloud global map is semantically segmented, with each point assigned to a corresponding label, and then the numpy index is used to extract the point cloud data corresponding to the label of interest. The training results show that the point cloud data classification reaches 53.7% on the public dataset SemanticKITTI and 77.4% on Semantic3D. The test on the SemanticKITTI-03 dataset reflects that our method is more efficient than traditional manual annotation on large-scale point cloud datasets. However, in order to save computing power, the algorithm network parameters in the point cloud classification part are too few, resulting in an unsatisfactory classification effect on small target objects. In the follow-up research, the classification of small target objects will be further optimized and tested.

**Author Contributions:** Conceptualization, J.L. (Jiadi Li); methodology, J.L. (Jiadi Li); validation, J.L. (Jiadi Li); formal analysis, Z.M. and J.L. (Jiajia Liu); investigation, Y.Z., Y.W. and J.Z.; project administration, Z.M.; writing—original draft, J.L. (Jiadi Li); writing—review and editing, Z.M. and J.L. (Jiajia Liu) All authors have read and agreed to the published version of the manuscript.

**Funding:** This paper is supported by the International Cooperation Project of Science and Technology Bureau of Chengdu (no. 2019-GH02-00051-HZ), Sichuan Unmanned System-Intelligent Perception, Engineering Laboratory Open Fund, and the research fund of the Chengdu University of Information Engineering, under grant nos. WRXT2020-001, WRXT2020-002, WRXT2021-002 and KYTZ202142. This paper is also supported by the Sichuan Science and Technology program of China, grant no. 2022YFS0565.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Acknowledgments:** The authors are grateful to the College of Automation, Chengdu University of Information Technology.

**Conflicts of Interest:** The authors declare that they have no conflict of interest to report regarding the present study.

#### **References**

**Tingyao Jiang, Cheng Li \* , Ming Yang and Zilong Wang**

College of Computer and Information, China Three Gorges University, Yichang 443002, China

**\*** Correspondence<EMAIL_ADDRESS>
**Abstract:** To improve the accuracy of the You Only Look Once v5s (YOLOv5s) algorithm for object detection, this paper proposes an improved YOLOv5s algorithm, CBAM-YOLOv5s, which introduces an attention mechanism. A convolutional block attention module (CBAM) is incorporated into the YOLOv5s backbone network to improve its feature extraction ability. Furthermore, the complete intersection-over-union (CIoU) loss is used as the object bounding-box regression loss function to accelerate the speed of the regression process. Experiments are carried out on the Pascal Visual Object Classes 2007 (VOC2007) dataset and the Microsoft Common Objects in Context (COCO2014) dataset, which are widely used for object detection evaluations. On the VOC2007 dataset, the experimental results show that compared with those of the original YOLOv5s algorithm, the precision, recall and mean average precision (mAP) of the CBAM-YOLOv5s algorithm are improved by 4.52%, 1.18% and 3.09%, respectively. On the COCO2014 dataset, compared with the original YOLOv5s algorithm, the precision, recall and mAP of the CBAM-YOLOv5s algorithm are increased by 2.21%, 0.88% and 1.39%, respectively.

**Keywords:** object detection; YOLOv5s; attention mechanism; deep learning

**Citation:** Jiang, T.; Li, C.; Yang, M.; Wang, Z. An Improved YOLOv5s Algorithm for Object Detection with an Attention Mechanism. *Electronics* **2022**, *11*, 2494. https://doi.org/ 10.3390/electronics11162494

Academic Editor: Stefanos Kollias

Received: 1 July 2022 Accepted: 9 August 2022 Published: 10 August 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

## **1. Introduction**

In recent years, due to the advent of the era of big data and the rapid development of computer graphics cards, the computing power of computers has also improved, which has accelerated the development of artificial intelligence in the computer field. There are more and more studies related to artificial intelligence, for example, the research in [1–4] has good application value, and the research of object detection has also developed accordingly.

Object detection has a wide range of applications in many areas of artificial intelligence, including robot navigation [5], autonomous driving [6], medical imaging [7] and human– object interaction [8]. Current object detection algorithms are mainly divided into singlestage detection algorithms and two-stage detection algorithms. Single-stage detection algorithms are represented by the You Only Look Once (YOLO) series [9–13], single-shot multibox detector (SSD) series [14–17], etc. Two-stage detection algorithms are represented by the region-based convolutional neural network (R-CNN) series [18–20]. A single-stage detection algorithm simultaneously classifies and locates the object of interest during object detection, while a two-stage detection algorithm performs these tasks separately. The characteristics of single-stage detection algorithms include that their detection speeds are very fast, but their accuracies are low. A two-stage detection algorithm is the opposite of a single-stage detection algorithm, with high accuracy but a slow detection speed. At present, most object detection tasks are real-time detection problems based on video, which require high detection speed, so a single-stage object detection algorithm is more suitable.

The latest single-stage object detection algorithm is the YOLOv5 algorithm. Compared with other single-stage object detection algorithms, the YOLOv5 algorithm has a faster detection speed and a smaller model. YOLOv5 is divided into four different algorithms: YOLOv5s, YOLOv5m, YOLOv5l and YOLOv5x. The network structures of these four

different algorithms are roughly the same, but their differences lie in the depths and widths of the networks. Among them, YOLOv5s has a faster detection speed and a smaller model than the other three algorithms, but the disadvantage is that its accuracy is low.

In recent years, the attention mechanism has been widely used in various fields of deep learning [21–24], including image processing, speech recognition and natural language processing. There are many attention mechanism modules in the field of computer vision, among which the most classic ones are the squeeze-and-excitation network (SENet) [23] and the convolutional block attention module (CBAM) [24]. SENet is the champion of the ImageNet2017 image recognition competition, and CBAM is the champion of the 2018 classification competition. In this paper, these two classical modules are introduced respectively for comparative experiments.

At present, there are many improved object detection models based on the attention mechanism, which have good results, but most of the model parameters are large, and the detection speed is not fast enough. The improved method in this paper has good performance in both detection effect and detection speed.

#### **2. CBAM-YOLOv5s**

In this section, the YOLOv5s algorithm is first introduced, followed by detailed descriptions of the improvements made to the YOLOv5s network structure and the object bounding-box regression loss function used by the algorithm proposed in this paper.

### *2.1. YOLOv5s Algorithm*

The YOLOv5s algorithm includes three parts: a feature extraction backbone network, a feature fusion neck network and a detection head. The network structure is shown in Figure 1. The detection process of the YOLOv5s algorithm is roughly divided into three steps. The first step is to extract features, adjust the scale of the input image to 640 × 640, and input the adjusted image into the backbone network. The BottleneckCSP-2 module, the BottleneckCSP-3 module, and the BottleneckCSP-4 module output three different scales of feature maps with sizes of 80 × 80, 40 × 40 and 20 × 20, respectively; these three feature maps contain different feature information. The second step is feature fusion. The three different scales of feature maps obtained through the backbone network are transmitted to the neck network, and the neck network performs a series of upsampling, convolution, channel concatenation and other operations to fully integrate the information provided by the feature maps. The third step is to output the detection heads. After the neck network fully integrates the features, three detection heads with sizes of 80 × 80, 40 × 40 and 20 × 20 are output. These three detection heads with different scales are used to detect small objects, medium objects and large objects.

Compared with YOLOv4, YOLOv5s adds a focus module to the backbone network. The main function of this module is to periodically extract pixels from high-resolution images and reconstruct them into low-resolution images to improve the receptive field of each pixel while retaining relatively complete original information. The design of the module is mainly used to reduce the number of calculations and speed up the algorithm. YOLOv4 only uses a cross-stage partial network (CSP) [25] structure in the backbone network, while YOLOv5s uses CSP structures in both the backbone network and the neck network. A CSP structure is used for local cross-layer network fusion, which reduces the number of calculations while simultaneously ensuring accuracy.

**Figure 1.** Structure of the YOLOv5s network. **Figure 1.** Structure of the YOLOv5s network.

#### *2.2. Improved YOLOv5s with an Attention Mechanism 2.2. Improved YOLOv5s with an Attention Mechanism*

An attention mechanism is a data processing method in that is widely used in various types of machine learning tasks, such as natural language processing, image recognition and speech recognition. An attention mechanism is essentially similar to the mechanism by which humans observe external objects; when humans observe external objects, they are first inclined to observe some important local information about these objects and then combine the information derived from different regions to form an overall impression of the observed objects. 2.2.1. CBAM An attention mechanism is a data processing method in that is widely used in various types of machine learning tasks, such as natural language processing, image recognition and speech recognition. An attention mechanism is essentially similar to the mechanism by which humans observe external objects; when humans observe external objects, they are first inclined to observe some important local information about these objects and then combine the information derived from different regions to form an overall impression of the observed objects.

> The CBAM is a lightweight module that includes a channel attention submodule and a spatial attention submodule. The channel attention submodule focuses on important

### 2.2.1. CBAM

feature information, and the spatial attention submodule focuses on object location information. The structure of the CBAM is shown in Figure 2. The CBAM is a lightweight module that includes a channel attention submodule and a spatial attention submodule. The channel attention submodule focuses on important feature information, and the spatial attention submodule focuses on object location information. The structure of the CBAM is shown in Figure 2. *Electronics* **2022**, *11*, x FOR PEER REVIEW 4 of 14

> The operation process of the channel attention submodule: The input feature map uses a global average pooling operation and a global maximum pooling operation to ag-

> The operation process of the spatial attention submodule: The output of the channel attention submodule is subjected to an average pooling operation and a maximum pooling operation to obtain a spatial attention tensor; this is followed by channel concatenation. Then, the spatial attention tensor is obtained through a convolution operation and the sigmoid activation function; finally, the spatial attention tensor is multiplied with the output of the channel attention submodule to obtain the output of the spatial attention

> The CBAM is incorporated into the backbone network of YOLOv5s, and the network structure is shown in Figure 3. The function of the module is to let the network know which part to focus on and to accordingly achieve prominent representations of important features while suppressing the less important features; this module can adjust the attention weight of the feature map and improve the feature extraction ability of the network.

> nel attention vector, sends it to a shared network, passes the added elements through a sigmoid activation function to obtain the resulting channel attention vector and finally multiplies the channel attention vector with the initial input to obtain the output of the

**Figure 2.** Structure of the CBAM. **Figure 2.** Structure of the CBAM.

channel attention submodule.

2.2.2. YOLOv5s Introduces the CBAM

submodule.

The operation process of the channel attention submodule: The input feature map uses a global average pooling operation and a global maximum pooling operation to aggregate the spatial information of the input feature map to obtain a one-dimensional channel attention vector, sends it to a shared network, passes the added elements through a sigmoid activation function to obtain the resulting channel attention vector and finally multiplies the channel attention vector with the initial input to obtain the output of the channel attention submodule.

The operation process of the spatial attention submodule: The output of the channel attention submodule is subjected to an average pooling operation and a maximum pooling operation to obtain a spatial attention tensor; this is followed by channel concatenation. Then, the spatial attention tensor is obtained through a convolution operation and the sigmoid activation function; finally, the spatial attention tensor is multiplied with the output of the channel attention submodule to obtain the output of the spatial attention submodule.

#### 2.2.2. YOLOv5s Introduces the CBAM

The CBAM is incorporated into the backbone network of YOLOv5s, and the network structure is shown in Figure 3. The function of the module is to let the network know which part to focus on and to accordingly achieve prominent representations of important features while suppressing the less important features; this module can adjust the attention weight of the feature map and improve the feature extraction ability of the network. *Electronics* **2022**, *11*, x FOR PEER REVIEW 5 of 14

**Figure 3.** Structure of the CBAM-YOLOv5s network. **Figure 3.** Structure of the CBAM-YOLOv5s network.

The specific operation of the CBAM is mainly divided into two steps, as shown in Figure 4. In the first step, the channel attention operation is performed on the input feature The specific operation of the CBAM is mainly divided into two steps, as shown in Figure 4.

to obtain an output of size 1024 × 20 × 20 with constant dimensions.

map. The input 1024 × 20 × 20 feature map is processed through a maximum pooling operation and an average pooling operation to obtain two 1024 × 1 × 1 feature maps, and then these two feature maps are each compressed by the first fully connected layer to

to output two 1024 × 1 × 1 feature maps. Then, the feature information of the two feature maps is added and passed through the sigmoid activation function to obtain a 1024 × 1 × 1 feature map, and finally, the feature map is multiplied by the initial input

In the second step, the spatial attention operation is performed. The 1024 × 20 × 20 feature map obtained through the channel attention operation is subjected to a maximum pooling operation and an average pooling operation to output two 1 × 20 × 20 feature maps, and then 2 × 20 × 20 feature maps are output through the channel concatenation operation. Next, the dimensions of the feature map are restored to 1 × 20 × 20 through a convolution operation, and this is followed by the sigmoid activation function, which outputs a 1 × 20 × 20 feature map. Then, the feature map is multiplied by the initial input to obtain a 1024 × 20 × 20 feature map, and this feature map is added to the input of the

In the first step, the channel attention operation is performed on the input feature map. The input 1024 × 20 × 20 feature map is processed through a maximum pooling operation and an average pooling operation to obtain two 1024 × 1 × 1 feature maps, and then these two feature maps are each compressed by the first fully connected layer to compress the number of channels to 64, thereby reducing the computational cost. This is followed by an expansion operation performed through the second fully connected layer to output two 1024 × 1 × 1 feature maps. Then, the feature information of the two feature maps is added and passed through the sigmoid activation function to obtain a 1024 × 1 × 1 feature map, and finally, the feature map is multiplied by the initial input to obtain an output of size 1024 × 20 × 20 with constant dimensions. *Electronics* **2022**, *11*, x FOR PEER REVIEW 6 of 14 BottleneckCSP-4 module to obtain the final output: a 1024 × 20 × 20 feature map. Finally, the extracted 1024 × 20 × 20 feature map is input back into the Conv-5 module in the neck network.

**Figure 4.** CBAM operation. **Figure 4.** CBAM operation.

*2.3. The Object Bounding-Box Regression Loss Function* The object bounding-box regression loss functions of most object detection algorithms use generalized intersection-over-union (*GIoU*) loss [26] to calculate the deviation between each prediction box and the corresponding ground truth; this loss is defined as = 1 − + | − | || (1) where *Ac* represents the area of the smallest box that contains both the ground truth and the prediction box, *IoU* represents the intersection over union of two bounding boxes, *U*  In the second step, the spatial attention operation is performed. The 1024 × 20 × 20 feature map obtained through the channel attention operation is subjected to a maximum pooling operation and an average pooling operation to output two 1 × 20 × 20 feature maps, and then 2 × 20 × 20 feature maps are output through the channel concatenation operation. Next, the dimensions of the feature map are restored to 1 × 20 × 20 through a convolution operation, and this is followed by the sigmoid activation function, which outputs a 1 × 20 × 20 feature map. Then, the feature map is multiplied by the initial input to obtain a 1024 × 20 × 20 feature map, and this feature map is added to the input of the BottleneckCSP-4 module to obtain the final output: a 1024 × 20 × 20 feature map. Finally,

The advantage of the *GIoU* loss is that it not only focuses on the overlapping area

the ground truth. However, the disadvantage of the *GIoU* loss is that when the ground truth or the prediction box surrounds the other, the *GIoU* loss function deteriorates, caus-

represents the union of the two bounding boxes and *LGIoU* represents the *GIoU* loss.

the extracted 1024 × 20 × 20 feature map is input back into the Conv-5 module in the neck network.

#### *2.3. The Object Bounding-Box Regression Loss Function*

The object bounding-box regression loss functions of most object detection algorithms use generalized intersection-over-union (*GIoU*) loss [26] to calculate the deviation between each prediction box and the corresponding ground truth; this loss is defined as

$$L\_{GIoII} = 1 - IoI + \frac{|Ac - \mathcal{U}|}{|Ac|} \tag{1}$$

where *Ac* represents the area of the smallest box that contains both the ground truth and the prediction box, *IoU* represents the intersection over union of two bounding boxes, *U* represents the union of the two bounding boxes and *LGIoU* represents the *GIoU* loss.

The advantage of the *GIoU* loss is that it not only focuses on the overlapping area between the prediction box and the ground truth but also focuses on other nonoverlapping areas, so it can better reflect the degree of overlap between the prediction box and the ground truth. However, the disadvantage of the *GIoU* loss is that when the ground truth or the prediction box surrounds the other, the *GIoU* loss function deteriorates, causing slow convergence and a large localization bias during the training process. The complete *IoU* (*CIoU*) loss [27] was developed in view of this problem; in addition to considering the overlapping area between the prediction box and the corresponding ground truth, the distance between the center points and the aspect ratio of the two bounding boxes are also considered. The *CIoU* loss is given as

$$L\_{CIoII} = 1 - IoI + \frac{\rho^2 \left(b\_\prime b^{gt}\right)}{c^2} + \varkappa v\_\prime \tag{2}$$

$$\alpha = \frac{v}{(1 - IoU) + v},\tag{3}$$

$$v = \frac{4}{\pi^2} \left( \arctan \frac{w^{\rm gt}}{h^{\rm gt}} - \arctan \frac{w}{h} \right)^2,\tag{4}$$

where *ρ* 2 (*b*, *b gt*) denotes the Euclidean distance between the prediction box and the ground truth, *c* denotes the shortest diagonal length of the smallest box containing both the ground truth and the prediction box, *α* is the weight parameter, *v* denotes the similarity between the aspect ratios of the two bounding boxes, *w gt* and *h gt* denote the width and height of the ground truth, *w* and *h* denote the width and height of the prediction box, respectively, and *LCIoU* denotes the *CIoU* loss.

Compared with the *GIoU* loss, the *CIoU* loss adds loss terms for the center distance and the aspect ratio between the prediction box and the ground truth to the loss function, which makes the prediction box converge faster and the regression localization more accurate, so the algorithm in this paper uses the *CIoU* loss as the object bounding-box regression loss function.

#### **3. Experiments**

To evaluate the improvement achieved by the CBAM introduced to YOLOv5s, the CBAM incorporated into the backbone network of YOLOv5s is replaced by another attention mechanism module called the SENet for an ablation experiment. In this section, the experimental equipment, dataset, evaluation metrics, experimental results and comparative analysis are introduced. We have put the core code of the algorithm on GitHub. Interested readers can download it, and the access link is https://github.com/2530525322/objectmodel (accessed on 9 August 2022).

The SENet mainly includes squeeze and excitation operations. The module structure is shown in Figure 5.

ing slow convergence and a large localization bias during the training process. The complete *IoU* (*CIoU*) loss [27] was developed in view of this problem; in addition to considering the overlapping area between the prediction box and the corresponding ground truth, the distance between the center points and the aspect ratio of the two bounding boxes are

> 2(, )

 (1−)+

ℎ −

*gt*) denotes the Euclidean distance between the prediction box and the ground

ℎ ) 2

2 + , (2)

, (3)

, (4)

*gt* denote the width and

= 1 − +

 = 4 2

between the aspect ratios of the two bounding boxes, *wgt* and *h*

=

(

truth, *c* denotes the shortest diagonal length of the smallest box containing both the ground truth and the prediction box, *α* is the weight parameter, *v* denotes the similarity

height of the ground truth, *w* and *h* denote the width and height of the prediction box,

Compared with the *GIoU* loss, the *CIoU* loss adds loss terms for the center distance and the aspect ratio between the prediction box and the ground truth to the loss function, which makes the prediction box converge faster and the regression localization more accurate, so the algorithm in this paper uses the *CIoU* loss as the object bounding-box re-

To evaluate the improvement achieved by the CBAM introduced to YOLOv5s, the CBAM incorporated into the backbone network of YOLOv5s is replaced by another attention mechanism module called the SENet for an ablation experiment. In this section, the experimental equipment, dataset, evaluation metrics, experimental results and comparative analysis are introduced. We have put the core code of the algorithm on GitHub. Interested readers can download it, and the access link is https://github.com/2530525322/ob-

The SENet mainly includes squeeze and excitation operations. The module structure

**Figure 5.** Structure of the SENet module. **Figure 5.** Structure of the SENet module.

ject-model (accessed on 9 August 2022).

also considered. The *CIoU* loss is given as

respectively, and *LCIoU* denotes the *CIoU* loss.

where *ρ*2

(*b*, *b*

gression loss function.

is shown in Figure 5.

**3. Experiments**

The SENet mainly focuses the network's attention on specific channels by learning the connections between channels, thereby achieving improved accuracy. The general The SENet mainly focuses the network's attention on specific channels by learning the connections between channels, thereby achieving improved accuracy. The general processing flow of the SENet is roughly divided into three steps.

processing flow of the SENet is roughly divided into three steps. Squeeze operation: The input W × H × C feature map is subjected to the global aver-Squeeze operation: The input W× H ×C feature map is subjected to the global average pooling operation to obtain a 1 × 1 × C feature map.

age pooling operation to obtain a 1 × 1 × C feature map. Excitation operation: The result of the squeeze operation is transformed nonlinearly Excitation operation: The result of the squeeze operation is transformed nonlinearly by using a fully connected layer.

by using a fully connected layer. Scale operation: The output obtained by the excitation operation is used as the weight and multiplied by the initial W × H × C input for the channel weights to obtain the final output.

#### *3.1. Experimental Equipment and Training Parameters*

The equipment used in the experiment is a Dell desktop computer, and its specific configuration is shown in Table 1.

**Table 1.** Computation system.

Some of the training parameters in the experiment are shown in Table 2.

#### *3.2. Dataset*

The datasets used in this experiment are the Pascal Visual Object Classes 2007 (VOC2007) dataset [28] and the Microsoft Common Objects in Context (COCO2014) dataset. The

COCO2014 dataset has a total of 123,287 images with 80 categories. The VOC2007 dataset contains a total of 9963 images. Twenty classes are included in the dataset, as shown in Figure 6; these classes include the airplane, bicycle, bird, boat, bottle, bus, car, cat, chair, cow, dining table, dog, horse, motorbike, person, potted plant, sheep, sofa, train and TV monitor categories, and the associated XML file provides the object class of the input image and the coordinates of the corresponding ground truth. *Electronics* **2022**, *11*, x FOR PEER REVIEW 9 of 14

**Figure 6.** VOC2007 dataset. **Figure 6.** VOC2007 dataset.

#### *3.3. Evaluation Metrics 3.3. Evaluation Metrics*

To evaluate the performance of the proposed algorithm, the evaluation metrics in this paper are precision (*P*), recall (*R*), mean average precision (mAP), F-score and frames per second (FPS). To evaluate the performance of the proposed algorithm, the evaluation metrics in this paper are precision (*P*), recall (*R*), mean average precision (mAP), F-score and frames per second (FPS).

Precision calculates the proportion of the number of correctly predicted positive samples to the total number of samples predicted as positive samples, that is, the accuracy of the prediction for the evaluation object. Precision is defined as follows: Precision calculates the proportion of the number of correctly predicted positive samples to the total number of samples predicted as positive samples, that is, the accuracy of the prediction for the evaluation object. Precision is defined as follows:

$$P = \frac{TP}{TP + FP'} \tag{5}$$

where *TP* represents true positives, that is, the number of positive samples predicted as positive samples; *FP* represents false positives, that is, the number of negative samples predicted as positive samples. where *TP* represents true positives, that is, the number of positive samples predicted as positive samples; *FP* represents false positives, that is, the number of negative samples predicted as positive samples.

Recall calculates the proportion of the number of correctly predicted positive samples to the total number of actual positive samples, that is, whether the evaluation object is completely found or not. Recall is defined as follows: Recall calculates the proportion of the number of correctly predicted positive samples to the total number of actual positive samples, that is, whether the evaluation object is completely found or not. Recall is defined as follows:

$$R = \frac{TP}{TP + FN} \tag{6}$$

, (7)

. (8)

negative samples. The *mAP* calculates the mean of the average precision (*AP*) values of all classes and is used to evaluate the overall performance of the algorithm. The *mAP* is given as where *FN* represents false negatives, that is, the number of positive samples predicted as negative samples.

=

 = ∫ 1 0

*F-score* calculates the harmonic value of precision and recall, which can comprehen-

∑ =1

sively measure these two indicators. The *F-score* is defined as follows:

The *mAP* calculates the mean of the average precision (*AP*) values of all classes and is used to evaluate the overall performance of the algorithm. The *mAP* is given as

$$AP = \int\_0^1 P \, d\mathcal{R}\_\prime \tag{7}$$

$$mAP = \frac{\sum\_{j=1}^{c} AP\_j}{c}.\tag{8}$$

*F-score* calculates the harmonic value of precision and recall, which can comprehensively measure these two indicators. The *F-score* is defined as follows:

$$F\text{-score} = \left(1 + \beta^2\right) \frac{P\cdot R}{\beta^2 \cdot P + R} \,\text{}\tag{9}$$

where *β* is used to balance the weight of precision and recall in the *F-score*, and there are three values. When *β* is equal to 1, precision is as important as recall; when *β* is less than 1, precision is more important than recall; when *β* is greater than 1, recall is more important than precision.

#### *3.4. Experimental Results and Comparative Analysis*

During the experimental training process, the stochastic gradient descent (SGD) [29] optimization algorithm is used to update the model parameters. Table 3 shows the experimental results obtained on the VOC2007 dataset.

**Table 3.** Ablation experiment.

As seen in Table 3, compared with those of the original YOLOv5s algorithm that does not introduce an attention mechanism, the precision, recall and mAP of the proposed algorithm that introduces an attention mechanism are improved. Compared with the original YOLOv5s, the YOLOv5s version with the SENet module achieves a 0.59% improvement in precision, a 1.22% improvement in recall and a 0.98% improvement in mAP, while the YOLOv5s version with the CBAM yields larger improvements, with a 4.52% improvement in precision, a 1.18% improvement in recall and a 3.09% improvement in mAP. By conducting a comparative analysis on the experimental results, it can be concluded that the algorithm in this paper has better performance than the original algorithm and the algorithm with the SENet module. SENet only includes channel attention and can only obtain important feature information on the channel, while CBAM includes not only channel attention but also spatial attention. It can obtain important feature information in both channel and space, so that the network can better learn important features in the image. The more picture features the network learns, the better it can recognize the object, which will make the network's recognition accuracy higher.

The experimental comparison results of the object bounding-box regression loss function are shown in Figure 7, where the horizontal axis is the number of epochs and the vertical axis is the value of the bounding-box loss. The experimental results show that the use of the CIoU loss as the bounding-box regression loss function results in faster convergence than the GIoU loss.

**Figure 7.** Variation in the two loss functions. **Figure 7.** Variation in the two loss functions. **Figure 7.** Variation in the two loss functions.

In order to further verify the effectiveness of the improved algorithm, this study includes comparative experiments on the COCO2014 dataset. The experimental results are shown in Table 4. In order to further verify the effectiveness of the improved algorithm, this study includes comparative experiments on the COCO2014 dataset. The experimental results are shown in Table 4. In order to further verify the effectiveness of the improved algorithm, this study includes comparative experiments on the COCO2014 dataset. The experimental results are

**Table 4.** Comparative experiment. **Table 4.** Comparative experiment.

shown in Table 4.

As can be seen from Table 4, compared with the original YOLOv5s algorithm, the precision, recall and mAP of the CBAM-YOLOv5s algorithm are increased by 2.21%, 0.88% and 1.39%, respectively. Based on the experimental results in Tables 3 and 4, it can be concluded that the improved CBAM-YOLOv5s algorithm is better than the original YOLOv5s algorithm on the VOC2007 dataset and the COCO2014 dataset. As can be seen from Table 4, compared with the original YOLOv5s algorithm, the precision, recall and mAP of the CBAM-YOLOv5s algorithm are increased by 2.21%, 0.88% and 1.39%, respectively. Based on the experimental results in Tables 3 and 4, it can be concluded that the improved CBAM-YOLOv5s algorithm is better than the original YOLOv5s algorithm on the VOC2007 dataset and the COCO2014 dataset. As can be seen from Table 4, compared with the original YOLOv5s algorithm, the precision, recall and mAP of the CBAM-YOLOv5s algorithm are increased by 2.21%, 0.88% and 1.39%, respectively. Based on the experimental results in Tables 3 and 4, it can be concluded that the improved CBAM-YOLOv5s algorithm is better than the original YOLOv5s algorithm on the VOC2007 dataset and the COCO2014 dataset.

Figure 8 shows the detection effect of the CBAM-YOLOv5s algorithm on the VOC2007 dataset. It can detect different targets in the picture and frame them. Figure 8 shows the detection effect of the CBAM-YOLOv5s algorithm on the VOC2007 dataset. It can detect different targets in the picture and frame them. Figure 8 shows the detection effect of the CBAM-YOLOv5s algorithm on the VOC2007 dataset. It can detect different targets in the picture and frame them.

**Figure 8.** The detection effect of the algorithm. **Figure 8.** The detection effect of the algorithm. **Figure 8.** The detection effect of the algorithm.

To verify the effect of the algorithm proposed in this paper, this paper also compares it with other object detection algorithms, as shown in Table 5. The precision and the FPS are used as measurement indicators.

**Table 5.** Comparison with other algorithms.

It can be seen from the results in Table 5 that the improved YOLOv5s performs better than the other detection algorithms in terms of precision and FPS. For example, it outperforms the YOLOv4 by 2.1% on the VOC2007 dataset with faster detection.

#### **4. Conclusions**

In this paper, a CBAM is incorporated into the backbone network of YOLOv5s to optimize its network structure, and the CIoU loss is used as the object bounding-box regression loss function to accelerate the speed of the regression process. To verify the performance of the proposed algorithm, extensive experiments are conducted on the VOC2007 dataset. The experimental results show that compared with those of the original YOLOv5s, the precision, recall and mAP of the proposed algorithm are significantly improved; furthermore, the CIoU loss is used because the bounding-box regression loss function is faster than the GIoU loss in terms of convergence. The algorithm in this paper solves the problem regarding the low detection accuracy of the original YOLOv5s algorithm to a certain extent, but the algorithm still exhibits certain detection errors and missed detection problems for complex images with dense objects. Future research will involve continuously optimizing the network structure of the proposed algorithm to further improve its detection accuracy.

**Author Contributions:** Conceptualization, T.J., C.L., M.Y. and Z.W.; data curation, T.J., C.L., M.Y. and Z.W.; methodology, T.J. and C.L.; writing—original draft, T.J. and C.L. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the National Natural Science Foundation of China, grant number 61871258.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

## *Article* **Recognition of Dorsal Hand Vein in Small-Scale Sample Database Based on Fusion of ResNet and HOG Feature**

**Jindi Li, Kefeng Li \* , Guangyuan Zhang \*, Jiaqi Wang, Keming Li and Yumin Yang**

School of Information Science and Electric Engineering, Shandong Jiaotong University, Jinan 250000, China

**\*** Correspondence<EMAIL_ADDRESS>(K.L<EMAIL_ADDRESS>(G.Z.)

**Abstract:** As artificial intelligence develops, deep learning algorithms are increasingly being used in the field of dorsal hand vein (DHV) recognition. However, deep learning has high requirements regarding the number of samples, and current DHV datasets have few images. To solve the above problems, we propose a method based on the fusion of ResNet and Histograms of Oriented Gradients (HOG) features, in which the shallow semantic information extracted by primary convolution and HOG features are fed into the residual structure of ResNet for full fusion and, finally, classification. By adding Gaussian noise, the North China University of Technology dataset, the Shandong University of Science and Technology dataset, and the Eastern Mediterranean University dataset are extended and fused to from a fused dataset. Our proposed method is applied to the above datasets, and the experimental results show that our proposed method achieves good recognition rates on each of the datasets. Importantly, we achieved a 93.47% recognition rate on the fused dataset, which was 2.31% and 26.08% higher than using ResNet and HOG alone.

**Keywords:** ResNet; HOG; feature fusion; DHV recognition

## **1. Introduction**

Biometric identification refers to a technology that uses the physiological or behavioral features of the human body (such as fingerprint features [1], face features [2], gait features [3], signature handwriting [4], etc.) to achieve identity authentication. Compared with the traditional authentication systems based on passwords, tokens, and certificates, biometric authentication systems have many advantages [5–7], so more and more people begin to focus on the research of biometric identification. As a kind of biometric identification method, dorsal hand vein recognition is different from other biometric identification methods. It mainly uses infrared light to collect images of the back of the hand and uses this method to show the outline structure of veins [8] in order to realize the identification of an individuals' identity. An anatomy article [9] has demonstrated that the DHV has a unique structure during growth and development, which can characterize the individual to a certain extent. Therefore, the research on DHV recognition is of great significance in individual recognition.

Currently, DHV recognition research is primarily focused on a single database, which makes it easier to achieve better recognition results due to the use of similar acquisition equipment, subjects, and collection environment. DHV identification on a single database includes two methods, namely traditional features and deep learning methods. In 2019, Vairavel et al. [10] studied the recognition performance of three classical dense descriptors for DHV recognition, including Local Binary Pattern(LBP), HOG, and Weber local descriptor (WLD), and achieved good results on the Northern University of Technology(NCUT) [11] database. Liu et al. [12] proposed an improved biometric map matching method in 2020, which achieved a 98.09% recognition rate on the Xi'an Jiaotong University (XJTU) database. With the rise of artificial intelligence, some researchers have begun to use deep learning methods for DHV identification. In 2019, Wang et al. [13] used the

**Citation:** Li, J.; Li, K.; Zhang, G.; Wang, J.; Li, K.; Yang, Y. Recognition of Dorsal Hand Vein in Small-Scale Sample Database Based on Fusion of ResNet and HOG Feature. *Electronics* **2022**, *11*, 2698. https://doi.org/ 10.3390/electronics11172698

Academic Editor: Dah-Jye Lee

Received: 27 July 2022 Accepted: 25 August 2022 Published: 28 August 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

selective convolution feature (SCF) model and spatial pyramid pooling (SPP) to obtain a more robust feature representation of images and they achieved excellent recognition rates on the China University of Mining and Technology (CUMT) databases. In 2019, Zhong et al. [14] designed a Deep Hashing Network (DHN) for DHV identification and achieved good results.

We can see from the above research that the DHV recognition of a single database has achieved good results, whether using traditional methods or deep learning methods. In recent years, researchers have begun to focus on cross-device DHV identification. In 2019, Wang et al. [15] proposed an improved scale-invariant feature transform (SIFT) algorithm, which achieved a recognition rate of 88.5% on datasets acquired by different devices by improving the scale factor *α*, extremum search neighborhood structure, and matching threshold R. In 2021, Wang et al. [16] proposed a two-stage coarse-to-fine matching method. First, the vein images to be matched are roughly matched in each category of the database, and then the SIFT method is used to extract the feature points of the vein images for fine matching. Such a method achieves good results on the cross-device DHV database.

Through the investigation and research on the cross-device DHV, most of the research on the cross-device DHV is based on the database of the same group of subjects and does not consider that the different subjects may have a certain impact on the experimental results. For different databases, there is not only the problem of different sampling equipment but also the diversity of subjects. Therefore, taking into account the differences between equipment and subjects is a major challenge in this field. In addition, in the cross-device DHV research, most researchers use traditional methods, but traditional methods are not robust to noise; if deep learning methods are used, they will face the problem of data volume. Given the above problems, this paper makes the following contributions:

#### **2. Materials and Methods**

The databases used in this paper are the dataset of the Shandong University of Science and Technology (SDUST) [17], the dataset of the Eastern Mediterranean University of Turkey (FYO) [18], the dataset of NCUT, and the fusion dataset (Fusion Dataset).

(1) SDUST Dataset

The dataset is a database of DHVs collected from the left and right hands of 63 males and 47 females using a commercial infrared device DF-300. The dataset contains 40 images of each subject, 20 for the left and right hands, for a total of 220 categories. The pictures in each category achieve image enhancement and data enhancement by changing brightness and random rotation, the size is 640 × 480 pixels, the horizontal and vertical resolutions are 96 dpi, and the format is jpg, as shown in Figure 1a.

(2) FYO Dataset

The FYO dataset was collected by a team from the Eastern Mediterranean University, which uses homemade equipment to collect data. The dataset collected images of the DHVs, palm veins, and wrist veins of the left and right hands of 160 volunteers (111 males and 49 females) twice, with a 10-min interval between the two acquisitions. The original data of the dataset contains data collected twice, each time there were 320 images of DHVs, 320 images of palm veins, and 320 images of wrist veins, and the images were all 800 × 600 color images, as shown in Figure 1b.

#### (3) NCUT Dataset This dataset builds a database of images of the backs of the hands of 102 people, in-

(3) NCUT Dataset

This dataset builds a database of images of the backs of the hands of 102 people, including 50 males and 52 females. During the collection, the left and right hands are alternately collected, that is, after collecting a vein picture with the left hand, the right hand is placed at the collection site to collect one image and then an image of the left hand is collected. This alternating method ensures the difference between the same type of samples. Due to the differences in the distribution of veins in the left and right hands of each individual, the database can be considered a back-of-hand image library composed of 204 types of samples. When collecting, 10 pictures are taken from the back of each hand, the size is 640 × 480 pixels, the horizontal and vertical resolutions are 96 dpi, the grayscale is 256 levels, and the format is bmp, as shown in Figure 1c. cluding 50 males and 52 females. During the collection, the left and right hands are alternately collected, that is, after collecting a vein picture with the left hand, the right hand is placed at the collection site to collect one image and then an image of the left hand is collected. This alternating method ensures the difference between the same type of samples. Due to the differences in the distribution of veins in the left and right hands of each individual, the database can be considered a back-of-hand image library composed of 204 types of samples. When collecting, 10 pictures are taken from the back of each hand, the size is 640 × 480 pixels, the horizontal and vertical resolutions are 96 dpi, the grayscale is 256 levels, and the format is bmp, as shown in Figure 1c.

and 49 females) twice, with a 10-min interval between the two acquisitions. The original data of the dataset contains data collected twice, each time there were 320 images of DHVs, 320 images of palm veins, and 320 images of wrist veins, and the images were all

*Electronics* **2022**, *11*, x FOR PEER REVIEW 3 of 20

800 × 600 color images, as shown in Figure 1b.

2.

0 0 adaptive

(1)

*x y*

( , )

(4) Fusion Dataset (4) Fusion Dataset

The fusion dataset includes the Shandong University of Science and Technology dataset, Northern University of Technology dataset, and Turkey Eastern Mediterranean University dataset, with a total of 372 volunteers and 744 sample data. Since the size and format of the images in different datasets are different, the images must be preprocessed first. First the images in the FYO dataset were transformed to grayscale and then their size was normalized, and the normalized size is 640 × 480 pixels. In this way, the images of the The fusion dataset includes the Shandong University of Science and Technology dataset, Northern University of Technology dataset, and Turkey Eastern Mediterranean University dataset, with a total of 372 volunteers and 744 sample data. Since the size and format of the images in different datasets are different, the images must be preprocessed first. First the images in the FYO dataset were transformed to grayscale and then their size was normalized, and the normalized size is 640 × 480 pixels. In this way, the images of the fusion dataset are all grayscale images with a size of 640 × 480 pixels, as shown in Figure 2. *Electronics* **2022**, *11*, x FOR PEER REVIEW 4 of 20

As can be seen from Figures 1 and 2, there are large differences in the images of dif-

This is mainly due to the differences in parameters, such as contrast, brightness, focal length, and optical performance of the lens between different acquisition devices, as well as the state of the collector's hand. These factors have a significant impact on the recognition results, and simple scale normalization is not conducive to extracting the texture features of the samples. Therefore, we need to extract the ROI of the image, and the method

method is used to determine the ROI area of the DHV image. The centroid of the vein

 ,

 *i j*

.

> ,

*j*

 *i j*

   ( , )

*j*

 ( , )

 *g i j*

 *g i*

 0  =

;

 *y*

> ( , ) *i j*

A square area with the size of R × R pixels is extracted and centered as the vein image to be processed. The experiment [21] verifies that when the ROI of the vein image is 380

**Figure 3.** ROI images from different datasets. (**a**) ROI of SDUST dataset. (**b**) ROI of FYO dataset. (**c**)

can be calculated as:

( , )

( , )

*g i j*

× 380, the recognition rate can achieve the best effect, as shown in Figure 3.

**Figure 2.** Fusion Dataset. **Figure 2.** Fusion Dataset.

2.1.2. Extract Image ROI

image expressed by

*g i j* ( , )

ROI of NCUT dataset.

where

*G x y*

( , )

of extracting ROI is studied in [19,20]. In this paper, the centroid

,

*i j*

,

*i g i j*

*i j*

0

=

is the grayscale value of pixel

*x*

#### 2.1.2. Extract Image ROI 2.1.2. Extract Image ROI

**Figure 2.** Fusion Dataset.

As can be seen from Figures 1 and 2, there are large differences in the images of different DHV databases, including changes in rotation angle, size, brightness, and noise. This is mainly due to the differences in parameters, such as contrast, brightness, focal length, and optical performance of the lens between different acquisition devices, as well as the state of the collector's hand. These factors have a significant impact on the recognition results, and simple scale normalization is not conducive to extracting the texture features of the samples. Therefore, we need to extract the ROI of the image, and the method of extracting ROI is studied in [19,20]. In this paper, the centroid (*x*0, *y*0) adaptive method is used to determine the ROI area of the DHV image. The centroid of the vein image expressed by *G*(*x*, *y*) can be calculated as: As can be seen from Figures 1 and 2, there are large differences in the images of different DHV databases, including changes in rotation angle, size, brightness, and noise. This is mainly due to the differences in parameters, such as contrast, brightness, focal length, and optical performance of the lens between different acquisition devices, as well as the state of the collector's hand. These factors have a significant impact on the recognition results, and simple scale normalization is not conducive to extracting the texture features of the samples. Therefore, we need to extract the ROI of the image, and the method of extracting ROI is studied in [19,20]. In this paper, the centroid 0 0 ( , ) *x y* adaptive method is used to determine the ROI area of the DHV image. The centroid of the vein image expressed by *G x y* ( , ) can be calculated as:

$$\mathbf{x}\_{0} = \frac{\sum\_{i,j} i \times \mathbf{g}(i,j)}{\sum\_{i,j} \mathbf{g}(i,j)};\\y\_{0} = \frac{\sum\_{i,j} i \times \mathbf{g}(i,j)}{\sum\_{i,j} \mathbf{g}(i,j)}\tag{1}$$

where *g*(*i*, *j*) is the grayscale value of pixel (*i*, *j*). where *g i j* ( , ) is the grayscale value of pixel ( , ) *i j* .

0

*x*

*Electronics* **2022**, *11*, x FOR PEER REVIEW 4 of 20

A square area with the size of R × R pixels is extracted and centered as the vein image to be processed. The experiment [21] verifies that when the ROI of the vein image is 380 × 380, the recognition rate can achieve the best effect, as shown in Figure 3. A square area with the size of R × R pixels is extracted and centered as the vein image to be processed. The experiment [21] verifies that when the ROI of the vein image is 380 × 380, the recognition rate can achieve the best effect, as shown in Figure 3.

**Figure 3.** ROI images from different datasets. (**a**) ROI of SDUST dataset. (**b**) ROI of FYO dataset. (**c**) ROI of NCUT dataset. **Figure 3.** ROI images from different datasets. (**a**) ROI of SDUST dataset. (**b**) ROI of FYO dataset. (**c**) ROI of NCUT dataset. *Electronics* **2022**, *11*, x FOR PEER REVIEW 5 of 20

#### 2.1.3. Add Gaussian Noise 2.1.3. Add Gaussian Noise

Gaussian noise is a kind of noise whose probability density function obeys normal distribution. The main function of Gaussian noise injection, as a data enhancement technique, is to add random Gaussian noise to samples to reduce overfitting during model training. Since there is only one image for each person in each dataset, it is not enough to prove the performance of the proposed method. Therefore, this method is adopted in this paper to expand the dorsal vein dataset. The dataset after adding Gaussian noise is shown in Figure 4. Gaussian noise is a kind of noise whose probability density function obeys normal distribution. The main function of Gaussian noise injection, as a data enhancement technique, is to add random Gaussian noise to samples to reduce overfitting during model training. Since there is only one image for each person in each dataset, it is not enough to prove the performance of the proposed method. Therefore, this method is adopted in this paper to expand the dorsal vein dataset. The dataset after adding Gaussian noise is shown in Figure 4.

gradient direction histograms of local regions of an image.

The acquisition of HOG features is divided into four steps:

*I x y I x y*

( , )

*x*

, and

( , )

*G x y*

*G x y H x*

( , ) ( , ) , ( 0.5)

and the gradient calculation formula are shown in Formulas (3) and (4).

*H x y* ( , )

> ( , ) *x y*

( , ) tan

*x y*

( , ) *x y*

=

=

=

> ( 1, )

 +

 ( , )

1

−

*x*

 *G x y*

=

( , ) ( , 1) ( , 1) *G x y H x y H x y y*

cessing for object detection, which constitutes a feature by computing and counting the

The first step is to normalize the color space of the DHV image, which consists of two aspects, image grayscale, and Gamma correction. Because our image is already a grayscale map, only Gamma correction is performed, and the Gamma correction formula is as

The gradient is calculated in the horizontal and vertical directions in the second step,

 *y H x*

 −

2

 +

= + − −

 =

> ( 1, )

> > ( , )

> > >

 *y*

 *G x y*

( , )

*y*

*G x y*

*G x y*

*x*

( , )

 −  *y*

are shown in Formulas (5) and (6).

 2

denote the horizontal gradient, vertical gradi-

in the input image, respectively. The amplitude

(2)

(3)

(4)

(5)

(6)

*2.2. Related Algorithms*

shown in Formula (2).

( , ) *G x y x*

, ( , )

ent, and pixel value at pixel point

*G x y*

and direction of the gradient at pixel

*y*

2.2.1. HOG

where

### *2.2. Related Algorithms*

#### 2.2.1. HOG

HOG [22] feature is a feature descriptor used in computer vision and image processing for object detection, which constitutes a feature by computing and counting the gradient direction histograms of local regions of an image.

The acquisition of HOG features is divided into four steps:

The first step is to normalize the color space of the DHV image, which consists of two aspects, image grayscale, and Gamma correction. Because our image is already a grayscale map, only Gamma correction is performed, and the Gamma correction formula is as shown in Formula (2).

$$I(\mathbf{x}, y) = I(\mathbf{x}, y)^{\gamma}, (\gamma = 0.5) \tag{2}$$

The gradient is calculated in the horizontal and vertical directions in the second step, and the gradient calculation formula are shown in Formulas (3) and (4).

$$G\_{\mathbf{x}}(\mathbf{x}, \mathbf{y}) = H(\mathbf{x} + \mathbf{1}, \mathbf{y}) - H(\mathbf{x} - \mathbf{1}, \mathbf{y}) \tag{3}$$

$$\mathbf{G}\_{\mathbf{y}}(\mathbf{x}, \mathbf{y}) = H(\mathbf{x}, \mathbf{y} + \mathbf{1}) - H(\mathbf{x}, \mathbf{y} - \mathbf{1}) \tag{4}$$

where *Gx*(*x*, *y*),*Gy*(*x*, *y*), and *H*(*x*, *y*) denote the horizontal gradient, vertical gradient, and pixel value at pixel point (*x*, *y*) in the input image, respectively. The amplitude and direction of the gradient at pixel (*x*, *y*) are shown in Formulas (5) and (6).

$$G(x,y) = \sqrt{G\_x(x,y)^2 + G\_y(x,y)^2} \tag{5}$$

The fourth step is to concatenate the HOG feature descriptors of all blocks in the im-

Before the idea of residual learning was proposed, traditional convolutional networks or fully connected networks had more or fewer problems, such as information loss and loss when information was transmitted. In addition, deep networks cannot be trained when gradients are small or exploding. ResNet [23] is a deep learning network that solves the problem of network degradation by introducing a deep residual learning framework. The network uses a residual unit structure, as shown in Figure 6. Assuming that the input

. The equation of

feedforward neural network with shortcut connections, and the residual unit structure can avoid the feature loss of the convolutional layer during the information transmission

*F x*

( ) +

and the residual unit of the learned feature can

can be implemented by a

 *x*

age to represent the HOG feature of the image, which is a 1 × 26,244 vector.

*H x*( )

 *x*

 −

$$a(\mathbf{x}, y) = \tan^{-1} \left( \frac{G\_y(\mathbf{x}, y)}{G\_x(\mathbf{x}, y)} \right) \tag{6}$$

The third step is to divide the image into 8 × 8 pixel cells. As shown in the red grid in Figure 5, a total of 784 cells are included, and the feature descriptors of each cell are counted. There are 9 descriptors for each cell, representing from 0◦ to 160◦ . Every 4 cells is a block, which is represented by the yellow grid in Figure 5. It contains a total of 729 blocks. The descriptors of all cells in each block are the HOG features of the block. Figure 5, a total of 784 cells are included, and the feature descriptors of each cell are counted. There are 9 descriptors for each cell, representing from 0° to 160°. Every 4 cells is a block, which is represented by the yellow grid in Figure 5. It contains a total of 729 blocks. The descriptors of all cells in each block are the HOG features of the block.

**Figure 5.** Divided into 28 × 28 cell images of veins. **Figure 5.** Divided into 28 × 28 cell images of veins.

feature is x, the learned feature is

*F x*

( )  *H x*

=

 ( )

2.2.2. ResNet Network

be represented as

**Figure 6.** Residual structure.

process.

The fourth step is to concatenate the HOG feature descriptors of all blocks in the image to represent the HOG feature of the image, which is a 1 × 26,244 vector. Before the idea of residual learning was proposed, traditional convolutional networks or fully connected networks had more or fewer problems, such as information loss

age to represent the HOG feature of the image, which is a 1 × 26,244 vector.

The fourth step is to concatenate the HOG feature descriptors of all blocks in the im-

#### 2.2.2. ResNet Network and loss when information was transmitted. In addition, deep networks cannot be trained

2.2.2. ResNet Network

**Figure 5.** Divided into 28 × 28 cell images of veins.

Before the idea of residual learning was proposed, traditional convolutional networks or fully connected networks had more or fewer problems, such as information loss and loss when information was transmitted. In addition, deep networks cannot be trained when gradients are small or exploding. ResNet [23] is a deep learning network that solves the problem of network degradation by introducing a deep residual learning framework. The network uses a residual unit structure, as shown in Figure 6. Assuming that the input feature is x, the learned feature is *H*(*x*) and the residual unit of the learned feature can be represented as *F*(*x*) = *H*(*x*) − *x*. The equation of *F*(*x*) + *x* can be implemented by a feedforward neural network with shortcut connections, and the residual unit structure can avoid the feature loss of the convolutional layer during the information transmission process. when gradients are small or exploding. ResNet [23] is a deep learning network that solves the problem of network degradation by introducing a deep residual learning framework. The network uses a residual unit structure, as shown in Figure 6. Assuming that the input feature is x, the learned feature is *H x*( ) and the residual unit of the learned feature can be represented as *F x H x x* ( ) ( ) = − . The equation of *F x x* ( ) + can be implemented by a feedforward neural network with shortcut connections, and the residual unit structure can avoid the feature loss of the convolutional layer during the information transmission process.

*Electronics* **2022**, *11*, x FOR PEER REVIEW 6 of 20

The third step is to divide the image into 8 × 8 pixel cells. As shown in the red grid in

Figure 5, a total of 784 cells are included, and the feature descriptors of each cell are counted. There are 9 descriptors for each cell, representing from 0° to 160°. Every 4 cells is a block, which is represented by the yellow grid in Figure 5. It contains a total of 729

blocks. The descriptors of all cells in each block are the HOG features of the block.

**Figure 6.** Residual structure. Figure 7 shows the residual structure of ResNet34, Table 1 shows the network parameters for the ResNet34. The main branch of the residual structure is composed of two

**Figure 6.** Residual structure.

Figure 7 shows the residual structure of ResNet34, Table 1 shows the network parameters for the ResNet34. The main branch of the residual structure is composed of two layers of 3 × 3 convolutional layers, and the connecting line on the right side of the residual structure is the shortcut branch, that is, the identity branch. Such branches are designed to reduce the amount of computation and parameters. layers of 3 × 3 convolutional layers, and the connecting line on the right side of the residual structure is the shortcut branch, that is, the identity branch. Such branches are designed to reduce the amount of computation and parameters.

**Table 1.** The ResNet34 network parameters in this paper. (\* represents the number of categories

**Layer Name Network Parameters Input Size Output Size** conv1 7 × 7, 64, stride2 224 × 224 × 3 56 × 56 × 3

,stride

,stride

,stride

Pool2d((H,W)) H = 1,W = 1 7 × 7 × 512 1 × 1 × 512

FC \ 512 \*

The framework based on the fusion of ResNet and HOG features proposed in this

First, we do two-way processing on the image input to the neural network, in which

we perform a convolution operation on it to extract the low-level semantic information of the image as a Feature Map. The other way is to input into the HOG function to extract the gradient information of the image. When performing HOG feature extraction on images, we make some changes to the features. First, we obtain the HOG feature of the entire

, stride2 56 × 56 × 64 56 × 56 × 64

56 × 56 × 64 28 × 28 × 128

28 × 28 × 128 14 × 14 × 256

14 × 14 × 256 7 × 7 × 512

**Figure 7.** ResNet34 residual structure. **Figure 7.** ResNet34 residual structure.

classified.).

Conv\_block1

Conv\_block2

Conv\_block3

Conv\_block4

AdaptiveAvg-

*2.3. Proposed Methods*

paper is shown in Figure 8.

 64

2

2

2

 64

 256

 256

3 3, 512

3 3 512

3 3, 128

3 3 128

3

4

6

3

3 3,

3 3

3 3,

2.3.1. Fusion of ResNet and HOG Feature

3 3

**Table 1.** The ResNet34 network parameters in this paper. (\* represents the number of categories classified.).

#### *2.3. Proposed Methods*

#### 2.3.1. Fusion of ResNet and HOG Feature

The framework based on the fusion of ResNet and HOG features proposed in this paper is shown in Figure 8.

First, we do two-way processing on the image input to the neural network, in which we perform a convolution operation on it to extract the low-level semantic information of the image as a Feature Map. The other way is to input into the HOG function to extract the gradient information of the image. When performing HOG feature extraction on images, we make some changes to the features. First, we obtain the HOG feature of the entire image according to the general process, and the extracted feature vector is a one-dimensional vector of 1 × 26,244, as shown in Equation (7).

$$V = [l\_1, l\_2, \dots, l\_{26244}] \tag{7}$$

Since the acquired image HOG feature dimension is large, the feature vector needs to be normalized. Otherwise, the image features are jerkier in gradient descent and the model has difficulty converging when the neural network is learning. Therefore, the obtained one-dimensional vector is first normalized, and the normalization formula is shown in Formula (8).

$$f\_{out} = \frac{\mathbf{x}\_i - \min(\mathbf{x})}{\max(\mathbf{x}) - \min(\mathbf{x})} \tag{8}$$

After the normalization is completed, the normalized feature vector is reshaped into a feature map of 162 × 162, as shown in Formula (9), to obtain the HOG feature.

$$Fault\\_HOG = \begin{bmatrix} l\_1 & \cdots & l\_{162} \\ \vdots & \ddots & \vdots \\ l\_{26082} & \cdots & l\_{26244} \end{bmatrix} \tag{9}$$

Since the size of HOG\_Map is different from that of Feature\_Map, a convolution operation is required. The convolved HOG feature is HOG\_Feature, and then spatial feature fusion is performed with Feature\_Map. The fusion method is shown in Equation (10), and the fusion method is shown in Formula (11). Then input the fused features into the ResNet residual block, and finally reduce the dimension of the feature map output by the ResNet residual block and input it into the fully connected layer for classification.

$$y\_{c,h,w}^{sum} = a \mathfrak{x}\_{c,h,w}^{a} + (1 - a)\mathfrak{x}\_{c,h,w}^{b} \tag{10}$$

*Electronics* **2022**, *11*, x FOR PEER REVIEW 8 of 20

sional vector of 1 × 26,244, as shown in Equation (7).

*Feature*\_*Fusion*\_*Map* = *α* × *HOG*\_*Feature* + (1 − *α*) × *Feature*\_*Map* (11) 1 2 26244  *l*[,, ,](7)

image according to the general process, and the extracted feature vector is a one-dimen-

 *l l*

=

*V*

**Figure 8.** ResNet and HOG feature fusion methods. (1.) Feature Fusion. This part obtains the HOG feature and shallow semantic information of the image, respectively, and then performs spatial feature fusion. (2.) Training and classification. The features after feature fusion are input into the residual architecture of ResNet for training, then input into the average pooling layer for dimensionality reduction, and finally input into the fully connected layer for classification. **Figure 8.** ResNet and HOG feature fusion methods. (1.) Feature Fusion. This part obtains the HOG feature and shallow semantic information of the image, respectively, and then performs spatial feature fusion. (2.) Training and classification. The features after feature fusion are input into the residual architecture of ResNet for training, then input into the average pooling layer for dimensionality reduction, and finally input into the fully connected layer for classification.

Since the acquired image HOG feature dimension is large, the feature vector needs 2.3.2. Feature Fusion Parameter Selection

to be normalized. Otherwise, the image features are jerkier in gradient descent and the model has difficulty converging when the neural network is learning. Therefore, the obtained one-dimensional vector is first normalized, and the normalization formula is shown in Formula (8).When dividing the image into cells, we performed three sets of experiments on the Twenty dataset of NCUT to verify the effect of the number of cells on feature fusion. The experimental results are shown in Table 2.

*i*

−

*x*

min( )

 *x*

**Table 2.** Different Cell identification results.

**Methods**

Res-Net\_HOG (ours)

As can be seen from Table 2, as the number of cells increases from 196 to 784, the feature fusion recognition rate gradually increases. However, as the number of cells increases, when reaching 3136 cells, the recognition rate not only does not increase but instead decreases. We deduce that the reason is that when the number of cells increases to a certain level, if the number of cells is increased, the gradient information of each cell will be lost to a certain extent, resulting in a decrease in the recognition effect. It can also be seen from Table 2 that as the number of cells increases, the model training time also increases. Considering the above two factors, we chose the number of cells to be 784.

*Electronics* **2022**, *11*, x FOR PEER REVIEW 10 of 20

#### **3. Experiments and Analysis**

#### *3.1. Feature Fusion Validity Experiments*

In this paper, the dataset is divided into a training set, validation set, and test set according to 8:1:1 by random division. In addition, to ensure the persuasiveness of the experimental results, we conducted each experiment three times on the test set and took the average value. We used the Pytorch deep learning framework. The graphics card was NVIDIA GeForce RTX 2080 Ti 16 GB, the batch size was 16, the learning rate was 0.001, the loss function is the cross entropy loss function, and the epochs were 50. can achieve a recognition rate of more than 90% on the Fifty dataset of a single database, but it does not achieve such a high effect on the Twenty dataset. This is because the larger the number of samples, the better the trained model will be, and the stronger the generalization ability of the model. Secondly, we can see from Table 3 that as the number of data

Before feature fusion, we performed nine experiments on NCUT's Twenty dataset to find the best fusion factor *α*, and the experimental results are shown in Figure 9. When the fusion factor is *α* = 0.3, the recognition rate can achieve the best effect. Therefore, the fusion factors of ResNet and HOG in the following experiments are both set to 0.3. increases, using the HOG algorithm cannot effectively improve the recognition rate, because the traditional method has no dependence on the amount of data in the dataset.

**Figure 9.** ResNet and HOG fusion factor take values. **Figure 9.** ResNet and HOG fusion factor take values.

In addition, it can be seen from the table that the recognition rate of our proposed feature fusion method is better than that of using ResNet and HOG alone, which proves the feasibility of our proposed feature fusion method. **Table 3.** ResNet and HOG feature fusion recognition rate. **Recognition Rate(%)** Table 3 shows the recognition rates on a single dataset using ResNet, HOG, and ResNet\_HOG methods. First of all, it can be seen from Table 3 that using the ResNet network can achieve a recognition rate of more than 90% on the Fifty dataset of a single database, but it does not achieve such a high effect on the Twenty dataset. This is because the larger the number of samples, the better the trained model will be, and the stronger the generalization ability of the model. Secondly, we can see from Table 3 that as the number of data increases, using the HOG algorithm cannot effectively improve the recognition rate, because the traditional method has no dependence on the amount of data in the dataset.

The fusion dataset is consistent with the experiments performed on the single dataset, and the experimental results are shown in Table 4. Table 4 shows the comparison between the proposed feature fusion method and the ResNet method, and it can be seen that the recognition rate of the proposed method is better than that of using ResNet and HOG

> HOG 67.34 67.36 67.37 67.39 ResNet 83.70 86.83 90.46 91.16

ResNet\_HOG (ours) **85.70 89.46 92.27 93.47**

**Twenty Thirty Forty Fifty**

**SDUST FYO NCUT Twenty Thirty Forty Fifty Twenty Thirty Forty Fifty Twenty Thirty Forty Fifty**

alone.

HOG 83.06 83.13 83.15 83.16 82.01 82.04 82.08 82.10 81.30 81.32 81.34 81.35 ResNet 84.97 90.47 90.86 92.30 86.60 89.59 91.93 93.43 83.93 87.70 89.13 90.06

**Table 4.** Recognition rate on the fused dataset after feature fusion.

**Methods Recognition Rate(%)**

**Table 3.** ResNet and HOG feature fusion recognition rate.

In addition, it can be seen from the table that the recognition rate of our proposed feature fusion method is better than that of using ResNet and HOG alone, which proves the feasibility of our proposed feature fusion method.

The fusion dataset is consistent with the experiments performed on the single dataset, and the experimental results are shown in Table 4. Table 4 shows the comparison between the proposed feature fusion method and the ResNet method, and it can be seen that the recognition rate of the proposed method is better than that of using ResNet and HOG alone.

**Table 4.** Recognition rate on the fused dataset after feature fusion.

#### *3.2. Feature Fusion Robustness Experiments*

The robustness of the model has always been the focus of cross-database dorsal vein recognition research, and traditional methods are not robust to cross-database dorsal vein images and datasets with Gaussian noise added. Here we conduct experiments on three different datasets using the HOG algorithm. In addition, we use the Partition Local Binary Patterns (PLBP) [24] algorithm for comparison. PLBP is an improvement based on the LBP algorithm. It divides an image into non-overlapping blocks, uses the LBP algorithm for each block, and finally splices the LBP feature statistical histograms of the entire image. The experimental results are shown in Table 5.

**Table 5.** Comparison of different methods.

It can be seen from Table 5 that the recognition rate of using the PLBP and HOG algorithms alone not only does not achieve good results but also is much lower than the experimental results of other researchers [10] on the NCUT dataset. The reason for the analysis is that when other researchers conducted experiments on the NCUT dataset, they used the original dataset and did not use Gaussian noise to expand the dataset. Additionally, our experiments are performed on the dataset augmented with Gaussian noise, which also shows that traditional features are not robust to our dataset with Gaussian

noise added. Furthermore, it can be seen from Table 5 that in the FYO dataset, the effect of using the PLBP algorithm is particularly low, and we find some categories with the worst recognition results in the FYO dataset, as shown in Figure 10. Most of the categories with poor recognition results are recognized in the category 52. Figure 11 shows a statistical histogram of the texture information extracted from the DHV images using LBP with rotationally invariant consistency pattern. As can be seen from the figure, the texture information for categories 131 and 170 differs significantly from the registered features but differs very little from the registered features for category 52, which leads to DHV images like 131 and 170 being easily identified as the category 52. Analysis of the reasons for this occurrence, by adding Gaussian noise leads to a variation in the intra-class images, where the intra-class distances become larger and are thus misidentified as other classes. *Electronics* **2022**, *11*, x FOR PEER REVIEW 12 of 20

170.

The HOG algorithm can achieve a recognition rate of more than 80% on a single database, but only 67.39% on a fusion dataset. We find some of the worst-recognized classes in the fused dataset, which are mostly data from NCUT and SDUST, as shown in Figure 12. Most of these partially identified worst classes are identified as 316 and 8, which are images in the FYO dataset, as shown in Figure 13.

We found that in the worst-recognized category, the blood vessel information of these images is not obvious, and the Gaussian noise on the images accounts for more. The images of the most misclassified categories have almost no blood vessel information, and most of the information is the hair on the back of the hand. The gap between these two categories cannot be seen visually. We visualize the HOG feature maps of these categories, as shown in Figure 14.

**Figure 11.** The misclassification of most categories of texture information. (**a**) Registration characteristics of category 52. (**b**) Registration characteristics of category 131. (**c**) Testing characteristics of category 131. (**d**) Registration characteristics of category 170. (**e**) Testing characteristics of category

**Figure 10.** The most misclassified tags in the FYO dataset.

**Figure 11.** The misclassification of most categories of texture information. (**a**) Registration characteristics of category 52. (**b**) Registration characteristics of category 131. (**c**) Testing characteristics of category 131. (**d**) Registration characteristics of category 170. (**e**) Testing characteristics of category 170. **Figure 11.** The misclassification of most categories of texture information. (**a**) Registration characteristics of category 52. (**b**) Registration characteristics of category 131. (**c**) Testing characteristics of category 131. (**d**) Registration characteristics of category 170. (**e**) Testing characteristics of category 170. in the fused dataset, which are mostly data from NCUT and SDUST, as shown in Figure 12. Most of these partially identified worst classes are identified as 316 and 8, which are images in the FYO dataset, as shown in Figure 13.

We found that in the worst-recognized category, the blood vessel information of these images is not obvious, and the Gaussian noise on the images accounts for more. The images of the most misclassified categories have almost no blood vessel information, and most of the information is the hair on the back of the hand. The gap between these two categories cannot be seen visually. We visualize the HOG feature maps of these categories,

**Figure 12.** Fusion dataset part identifies the worst class. **Figure 12.** Fusion dataset part identifies the worst class.

**Figure 13.** The most misclassified categories.

as shown in Figure 14.

**Methods**

Res-

The HOG algorithm can achieve a recognition rate of more than 80% on a single database, but only 67.39% on a fusion dataset. We find some of the worst-recognized classes in the fused dataset, which are mostly data from NCUT and SDUST, as shown in Figure 12. Most of these partially identified worst classes are identified as 316 and 8, which are

images in the FYO dataset, as shown in Figure 13.

**Figure 12.** Fusion dataset part identifies the worst class.

**Figure 14.** HOG feature visualization for misclassification. **Figure 14.** HOG feature visualization for misclassification.

As can be seen from Figure 14, the images with insignificant vein information in the NCUT and SDUST datasets and the images of these two categories in the FYO dataset have less obvious HOG features. Therefore, errors are prone to occuring when calculating the Euclidean distance between two categories, resulting in a low recognition rate using As can be seen from Figure 14, the images with insignificant vein information in the NCUT and SDUST datasets and the images of these two categories in the FYO dataset have less obvious HOG features. Therefore, errors are prone to occuring when calculating the Euclidean distance between two categories, resulting in a low recognition rate using the HOG algorithm on the fusion dataset.

the HOG algorithm on the fusion dataset. The recognition rate of the ResNet\_HOG method proposed in this paper is significantly higher than that of using ResNet and HOG alone on a single dataset. Moreover, we The recognition rate of the ResNet\_HOG method proposed in this paper is significantly higher than that of using ResNet and HOG alone on a single dataset. Moreover, we also achieved good recognition rates on the fusion dataset and have strong robustness.

#### also achieved good recognition rates on the fusion dataset and have strong robustness. *3.3. Comparison with Other Researchers*

*3.3. Comparison with Other Researchers* Recently, some researchers [25] achieved the current optimal results on the NCUT Recently, some researchers [25] achieved the current optimal results on the NCUT dataset using CNN and PLBP feature fusion. We adopted this idea and fused PLBP with ResNet for feature fusion, and the experimental results are shown in Table 6. As can be

dataset using CNN and PLBP feature fusion. We adopted this idea and fused PLBP with ResNet for feature fusion, and the experimental results are shown in Table 6. As can be

SDUST dataset, but not on the FYO and NCUT. In [18], the authors used a decision-level fusion of palm, dorsal, and wrist biometric features on vein images, which can make full use of hand biometric features, so this method is superior to our proposed method. In [26], the authors used the principal component analysis (PCA) method to expand the DHV dataset to 250 per category, which far exceeded our data volume, so their experimental results were superior to ours. However, when we expand our dataset to 250 per category,

the recognition rate is 99.93**%**, and the recognition results are superior to [26].

**Table 6.** Recognition rates of different feature fusion methods on a single database.

**Recognition Rate(%) SDUST FYO NCUT Twenty Thirty Forty Fifty Twenty Thirty Forty Fifty Twenty Thirty Forty Fifty**

Net\_PLBP 85.40 90.87 91.77 92.76 90.27 91.77 94.17 95.20 85.96 89.73 90.17 92.50

seen from Table 6, our proposed method achieves the current optimal results on the SDUST dataset, but not on the FYO and NCUT. In [18], the authors used a decision-level fusion of palm, dorsal, and wrist biometric features on vein images, which can make full use of hand biometric features, so this method is superior to our proposed method. In [26], the authors used the principal component analysis (PCA) method to expand the DHV dataset to 250 per category, which far exceeded our data volume, so their experimental results were superior to ours. However, when we expand our dataset to 250 per category, the recognition rate is 99.93%, and the recognition results are superior to [26].

**Table 6.** Recognition rates of different feature fusion methods on a single database.

Table 7 shows the comparison between our proposed feature fusion method and the current methods of cross-database, from which it can be seen that our method can achieve better results, except for [16]. The reason is that [15,16,28] use datasets collected through different devices and the same subjects, whereas we use datasets with different devices, different subjects, and different ethnicities, and these different factors have a significant impact on DHV identification [29], so our method is slightly below [16].

*3.4. Comparison between Feature Fusion and Data Volume*

Through the experiments in the previous sections, we can see that increasing the number of samples in the dataset can improve the recognition rate of the dorsal vein of hand, but of the two methods, feature fusion achieved better results in the recognition of dorsal vein in small samples. Figure 15 shows the relationship between feature fusion and the number of samples.

**Figure 15.** Data volume vs. feature fusion. (**a**) Results of ResNet, HOG, and ResNet\_HOG on FYO dataset. (**b**) Results of ResNet, HOG, and ResNet\_HOG on SDUST dataset. (**c**) Results of ResNet, HOG, and ResNet\_HOG on NCUT dataset. (**d**) Results of ResNet, HOG, and ResNet\_HOG on Fusion dataset. **Figure 15.** Data volume vs. feature fusion. (**a**) Results of ResNet, HOG, and ResNet\_HOG on FYO dataset. (**b**) Results of ResNet, HOG, and ResNet\_HOG on SDUST dataset. (**c**) Results of ResNet, HOG, and ResNet\_HOG on NCUT dataset. (**d**) Results of ResNet, HOG, and ResNet\_HOG on Fusion dataset.

**4. Discusion** *4.1. Gamma Value Influence* When a picture appears too bright or too dark, it leads to poor image contrast, and that is when Gamma correction needs to be performed. Our dataset was grayed out for the experiments, so the color information did not affect the experiments much. The Gamma correction of the images is required to make the black areas of the images appear brighter. When Gamma < 1, in the high gray value area, the dynamic range becomes smaller, the image contrast decreases, the overall gray value of the image becomes larger, From Figure 15, we can first see that our proposed feature fusion method achieved good results on the small sample DHV dataset. Secondly, we can see that when the amount of data reaches 40 pieces per category, the proposed feature fusion method exceeds the ResNet recognition rate of 50 pieces without feature fusion. This is because the shallow semantic information of the image is extracted using ResNet and then fused with HOG features; after a series of convolution operations, the final features for classification include both the deep semantic information and the gradient information of the image. Through such a feature fusion method, the features of the image can be fully obtained and thus be accurately classified.

and the image becomes brighter. When Gamma > 1, gamma >1 in the low gray value area,

#### the dynamic range becomes smaller, the image contrast decreases, and the overall gray **4. Discusion**

#### value of the image becomes smaller and darker. To find the appropriate value for Gamma *4.1. Gamma Value Influence*

< 1, we conducted four sets of experiments on three different datasets, and the experimental results are shown in the Table 8. From the table, we can see that the value of Gamma does affect different datasets, but overall, the effect of Gamma on the experimental results is minimal, and in most of the literature, Gamma is generally taken as 0.5, so in this paper, we also take 0.5. **Table 8.** The influence of different gamma values on the experiment. **Dataset Gamma Value 0.2 0.4 0.5(ours) 0.6 0.8** NCUT 86.52 86.61 86.53 86.58 86.60 FYO 90.26 90.35 90.46 90.38 90.42 SDUST 86.59 86.51 86.57 86.44 86.49 When a picture appears too bright or too dark, it leads to poor image contrast, and that is when Gamma correction needs to be performed. Our dataset was grayed out for the experiments, so the color information did not affect the experiments much. The Gamma correction of the images is required to make the black areas of the images appear brighter. When Gamma < 1, in the high gray value area, the dynamic range becomes smaller, the image contrast decreases, the overall gray value of the image becomes larger, and the image becomes brighter. When Gamma > 1, gamma >1 in the low gray value area, the dynamic range becomes smaller, the image contrast decreases, and the overall gray value of the image becomes smaller and darker. To find the appropriate value for Gamma < 1, we conducted four sets of experiments on three different datasets, and the experimental results are shown in the Table 8. From the table, we can see that the value of Gamma does affect different datasets, but overall, the effect of Gamma on the experimental results is minimal, and in most of the literature, Gamma is generally taken as 0.5, so in this paper, we also take 0.5.

*4.2. Influence of Gaussian Noise Intensity Model*

**Table 8.** The influence of different gamma values on the experiment.

#### *4.2. Influence of Gaussian Noise Intensity Model*

How the SNR affects the recognition rate of ResNet, we have carried out three sets of experiments on the Twenty dataset of NCUT, as shown in Table 9. In the table, we can see that the recognition rate of ResNet gets lower and lower as the noise increases. This is mainly because, when adding too much Gaussian noise, the vein information on the image is completely covered by the noise, which makes the model difficult to train and the recognition rate decreases.

**Table 9.** Influence of Gaussian noise intensity model.

### *4.3. Influence of Convolution Blocks*

We performed a separate convolution operation before inputting the image and HOG features into the ResNet residual structure, and this operation had an effect on the experimental results, which we performed on three different datasets, and the experimental results are shown in Table 10. From the Table 10, we can see that adding the convolutional block performs significantly on the NCUT and SDUST datasets but not on the FYO dataset, although the recognition rate with the convolutional block is better than that without the convolutional block. This is because without adding the convolutional block, reshaping the HOG feature size will lead to the loss of most of the HOG features, which will affect the recognition results.

**Table 10.** Experimental comparison with and without convolution blocks.

#### *4.4. The Influence of Increasing the Amount of Data HOG on the Model*

Through the experiments in Section 3, we can see that ResNet\_HOG outperforms ResNet alone for small sample dorsal hand vein recognition, but the gain obtained using ResNet\_HOG always seems to be around 1–3% regardless of the dataset size, and we analyze the reasons for this as follows.

In this paper, traditional features play an auxiliary role. Traditional features can extract information that cannot be obtained by depth features (such as gradient information, etc.), but this information has a limited impact on depth features, so the recognition results of the model are not significantly improved after performing feature fusion. We also found this problem in [17], where the authors used a fusion of ResNet and LBP features and only achieved a 1.97% higher recognition rate than using ResNet alone. In addition, we conducted four sets of experiments on the NCUT dataset to determine how much HOG affects ResNet. As can be seen from the experiments in the Table 11, the effect of feature

fusion seems to become less and less pronounced as the amount of data increases. This is because as the amount of data increases, ResNet has enough samples for training and the trained model becomes more and more robust, so it is possible to obtain a high recognition rate using only ResNet.

**Table 11.** Experimental comparison of different data volumes.

For the above analysis, we can see that either ResNet and HOG feature fusion or ResNet with LBP for feature fusion can outperform the recognition rate obtained using ResNet alone. However, because the recognition rate obtained using ResNet alone is more than 90%, the model's recognition rate is only improved by 1–3% after feature fusion, which is also effective for a dataset with few samples.

#### *4.5. Other Deep Learning vs. Traditional Methods Discussion*

In [17], the authors conducted experiments using the ResNet network. The authors performed two types of data enhancement (increasing the amount of data and changing the image brightness), and the recognition rate obtained by both enhancement methods was better than that obtained by using ResNet alone. In addition, the authors also performed the method of ResNet and LBP feature fusion, and the recognition rate after performing data enhancement using ResNet was 90.31%, while the recognition rate after performing ResNet and LBP feature fusion was 92.28%, which is an improvement but not significant. Our analysis shows that the black background information of the rotated image has an impact on the extraction of LBP features, which leads to the insignificant improvement of recognition results after feature fusion.

We searched many references and found no literature combining DL and HOG, but there are experiments with CNN and PLBP feature fusion. In [25], the authors designed three methods of CNN and PLBP feature fusion, namely serial fusion, decision fusion, and feature fusion. The decision fusion and feature fusion are the best, which are 0.34% higher than the CNN network without fusion.

According to the preceding literature, traditional features and deep learning for feature fusion not only excel in a few sample dorsal hand vein recognition but also improve the recognition rate of large sample data, demonstrating the feasibility of traditional features and deep learning for feature fusion.

#### **5. Conclusions**

In this paper, we design a method for the fusion of ResNet and HOG features, which achieves better results on small sample datasets. We adopt the methods of other researchers and conduct experiments on our dataset, and the experimental results show that the recognition rate of the feature fusion method is better than that of using ResNet alone. This proves that the combination of deep learning and traditional features can not only solve the problem that deep learning has a low recognition rate for small samples but also solve the problem that traditional features are not robust to Gaussian noise. It further illustrates the superiority and feasibility of using deep learning and traditional feature fusion in the field of DHV recognition.

At present, our work has achieved good results on the dataset with Gaussian noise. In the future, we will utilize more ways to expand the dataset for verification, such as physical expansion (random rotation, image translation, image exposure, etc.) and deep learning automatically expansion [30,31]. Our fusion dataset now includes three different datasets, and we hope to obtain more datasets in the future to expand the database of DHVs. In addition, to explore the possibility of feature fusion between deep features and traditional features, we will use a variety of traditional features and deep feature fusion methods to verify the DHV dataset.

**Author Contributions:** Investigation, J.L.; data curation, K.L. (Keming Li), J.W. and Y.Y.; software, K.L. (Kefeng Li) and G.Z.; writing—original draft preparation, J.L. and K.L. (Kefeng Li); writing review and editing, K.L. (Kefeng Li). All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** The study was conducted in accordance with the Declaration of Helsinki, and approved by the The Ethics Committee of Shandong Jiaotong University(protocol code<PHONE_NUMBER>893 and date of approval 27 July 2022).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study. Written informed consent has been obtained from the participants to publish this paper.

**Data Availability Statement:** The datasets used in this paper are all available on request from the authors of the following three articles. (1) **SDUST:** [17]. (2) **FYO:** [18]. (3) **NCUT:** [11].

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

## *Article* **Research on Path-Planning Algorithm Integrating Optimization A-Star Algorithm and Artificial Potential Field Method**

**Lisang Liu 1,2 , Bin Wang 1,2,\* and Hui Xu 1,2**

**Abstract:** A fusion pathfinding algorithm based on the optimized A-star algorithm, the artificial potential field method and the least squares method is proposed to meet the performance requirements of path smoothing, response speed and computation time for the path planning of home cleaning robots. The fusion algorithm improves the operation rules of the traditional A-star algorithm, enabling global path planning to be completed quickly. At the same time, the operating rules of the artificial potential field method are changed according to the path points found by the optimal A-star algorithm, thus greatly avoiding the dilemma of being trapped in local optima. Finally, the least squares method is applied to fit the complete path to obtain a smooth path trajectory. Experiments show that the fusion algorithm significantly improves pathfinding efficiency and produces smoother and more continuous paths. Through simulation comparison experiments, the optimized A-star algorithm reduced path-planning time by 60% compared to the traditional A-star algorithm and 65.2% compared to the bidirectional A-star algorithm path-planning time. The fusion algorithm reduced the path-planning time by 65.2% compared to the ant colony algorithm and 83.64% compared to the RRT algorithm path-planning time.

**Keywords:** A-star algorithm; artificial potential field method; least squares method; path planning

## **1. Introduction**

Thanks to the rapid development of artificial intelligence technology, cleaning robots are now being introduced into ordinary households. One of the key parts of a cleaning robot is the path planning of the cleaning area [1,2]. Path-planning technology involves following certain pathfinding rules in an environment with obstacles to obtain a collision-free path from the starting point to the target point that satisfies the evaluation metrics [3,4]. As the complexity of the working environment of mobile robots continues to increase, it also places higher demands on path-planning techniques. Depending on the environment in which the robot works and the work requirements, it can be divided into global path planning and local path planning [5,6]. Global path planning can be divided into graph-based search algorithms, sampling-based planning algorithms, biomimetic-based algorithms, neural network algorithms, etc. Local path planning is divided into the dynamic window method, time-elastic band method, artificial potential field method, etc.

In 1959, Dijkstra, a Dutch scientist, was the first to propose an algorithm to solve the single-source shortest path problem, which was one of the earliest global path-planning algorithms [7]. The algorithm is centered on the starting point using a breadth-first search strategy to continuously expand outwards and then continuously search for the shortest path between the starting point and each expanded node in the map until it finds the target node, completing the path planning. In 1968, P. Hart, N. Nilsson and B. Raphael first proposed the heuristic A-star algorithm to solve the global path optimal problem [8]. The traditional A-star algorithm starts from the starting point and calculates the cost of

**Citation:** Liu, L.; Wang, B.; Xu, H. Research on Path-Planning Algorithm Integrating Optimization A-Star Algorithm and Artificial Potential Field Method. *Electronics* **2022**, *11*, 3660. https://doi.org/ 10.3390/electronics11223660

Academic Editor: Akshya Swain

Received: 10 October 2022 Accepted: 4 November 2022 Published: 9 November 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

moving the current node to the starting point and the ending point under the constraint of the evaluation function and extends radially to the target point. When an obstacle is encountered, it returns to the vicinity of the starting point to resume pathfinding and repeats until the target point is reached. Therefore, the algorithm generates a large number of useless nodes to be computed in the process of application, which leads to problems such as too much computation, too much memory occupation and too long pathfinding time. X. Zhang [9] optimized the algorithm by introducing a time factor to the A-star algorithm and combining it with a time window and priority strategy. Although this method reduces the number of turns and improves the efficiency of the system, it greatly increases the computational effort, and the chosen obstacle avoidance strategy tends to cause the algorithm to fall into a dead loop.

The artificial potential field method was proposed by Khatib in 1986. The idea is to simulate the environment in which the mobile robot is located as the "gravitational force" in physics, called a virtual potential field [10]. A virtual artificial potential field is formed by the repulsive field of an obstacle and the gravitational field of the target location, in which the mobile robot is influenced by the potential field to automatically search for a suitable collision-free path. As the robot moves, the potential field it is subjected to varies continuously, along a gradient from the repulsive field of a particular obstacle or the gravitational field of a target point alone. Due to its real-time nature, the artificial potential field method has been applied to the field of dynamic obstacles by many scholars. However, for scenarios where multiple obstacles exist at the same time, the problem of becoming caught in local minima that cannot be dislodged and the phenomenon of oscillation near the target point easily occur. Y. Wang et al. [11] addressed the problem that the potential field method tends to fall into local optima by improving the gravitational formulation of the traditional potential field method by adding new variables and also expanding the obstacle to a circular obstacle. The optimized algorithm solves the problem of the manual potential field method not being able to avoid large obstacles, greatly reduces the scanning time and reduces the working cost of the mobile robot. However, in an environment with irregular obstacles, it easily divides the passable paths between obstacles into obstacle regions, thus failing to obtain the optimal path. When the map is updated quickly, its real-time obstacle avoidance capability will be greatly reduced. H. Liu et al. [12] introduced the idea of fuzzy control into the path planning of mobile robots, dividing the environment in which the robot moves into two parts: a global safety region and a local danger, according to the location of obstacles and their influence range. In safe areas, the artificial potential field method acts on the robot to guide it towards the target; in dangerous areas, the artificial potential field method is combined with fuzzy control to guide the robot to avoid obstacles and move towards the target. The precise control of the deflection angle of the mobile robot effectively reduces the problem of unreachable targets and local minima. However, the fusion algorithm is limited in its application, and in dynamic environments, it relies mainly on the artificial potential field method, which does not practically solve the shortcomings of the artificial potential field method in dynamic environments.

This paper first optimizes the structure of the traditional A-star algorithm, then optimizes the potential field method by adding an intermittent point search strategy and constructing an intermittent point judgment function, and then combines the artificial potential field method and the least squares method to propose a fused path-planning algorithm. On the one hand, the structure of the traditional A-star algorithm is optimized to improve its speed in global path planning, and on the other hand, the artificial potential field method is optimized to solve the problem of the potential field method tending to fall into local optimality. The fusion algorithm overcomes the drawbacks of the original algorithm well and improves the efficiency and success rate of path planning. The paper concludes with a comparative simulation analysis of the optimized A-star algorithm and the traditional and bidirectional A-star algorithms for different starting points and different map environments. The fusion algorithm is analyzed and compared with the ant colony

algorithm and the RRT algorithm. The simulation comparison and data analysis confirm the fast, robust and advanced nature of the optimized fusion algorithm. **2. Global Path Planning** 103

the traditional and bidirectional A-star algorithms for different starting points and differ- 99 ent map environments. The fusion algorithm is analyzed and compared with the ant col- 100 ony algorithm and the RRT algorithm. The simulation comparison and data analysis con- 101 firm the fast, robust and advanced nature of the optimized fusion algorithm. 102

*Electronics* **2022**, *11*, x FOR PEER REVIEW 3 of 22

#### **2. Global Path Planning** *2.1. Traditional A-Star Algorithm* 104

#### *2.1. Traditional A-Star Algorithm* The A-star algorithm is a heuristic search algorithm for finding optimal paths in static 105

The A-star algorithm is a heuristic search algorithm for finding optimal paths in static obstacle environments [13]. It combines the advantages of Dijkstra's algorithm to find the shortest path well and the heuristic search algorithm breadth-first search (BFS) to search upwards at the most probable places first [14], to which a cost evaluation function is added to find the optimal path point. The principle is shown in Figure 1. obstacle environments [13]. It combines the advantages of Dijkstra's algorithm to find the 106 shortest path well and the heuristic search algorithm breadth-first search (BFS) to search 107 upwards at the most probable places first [14], to which a cost evaluation function is added 108 to find the optimal path point. The principle is shown in Figure 1. 109

**Figure 1.** Schematic diagram of the traditional A-star algorithm. (**a**) Conventional A-star algorithm 110 pathfinding in the absence of obstacles; (**b**) conventional A-star algorithm pathfinding in the pres- 111 ence of obstacles. 112 **Figure 1.** Schematic diagram of the traditional A-star algorithm. (**a**) Conventional A-star algorithm pathfinding in the absence of obstacles; (**b**) conventional A-star algorithm pathfinding in the presence of obstacles.

As shown in Figure 1a, S is the starting point and E is the end point. If the path en- 113 counters an obstacle, the algorithm returns to the starting point and searches again until 114 the search width exceeds the width of the obstacle, then continues to search and so on 115 until it reaches the target point. In practice, only a small number of nodes are relevant to 116 the path, but many nodes need to be computed. For this reason, a large number of useless 117 nodes are searched, creating problems such as too many calculations, more useless 118 memory usage and longer pathfinding times. In addition, there are too many turning 119 points in the planned path. The traditional A-star algorithm does not smooth the path, so 120 As shown in Figure 1a, S is the starting point and E is the end point. If the path encounters an obstacle, the algorithm returns to the starting point and searches again until the search width exceeds the width of the obstacle, then continues to search and so on until it reaches the target point. In practice, only a small number of nodes are relevant to the path, but many nodes need to be computed. For this reason, a large number of useless nodes are searched, creating problems such as too many calculations, more useless memory usage and longer pathfinding times. In addition, there are too many turning points in the planned path. The traditional A-star algorithm does not smooth the path, so the planned path turns rigidly, and the robot needs to accelerate and decelerate frequently to perform the turning action when walking, which is not conducive to the robot's path tracking.

the planned path turns rigidly, and the robot needs to accelerate and decelerate frequently 121 The set of optimal path points then forms the optimal path, where the cost evaluation function is as follows:

> f n hn

$$f(n) = h(n) + g(n) \tag{1}$$

tracking. 123 The set of optimal path points then forms the optimal path, where the cost evaluation 124 function is as follows: 125

> g n

() () () = + (1)

where

f n

where *f*(*n*) is the cost function of the current position, *g*(*n*) is the actual cost of the mobile robot from the starting point to the current position and *h*(*n*) is the estimated cost of the mobile robot from the current position to the position of the target point [15]. () ( ) ( ) = − +− 2 2 t n tn hn X X Y Y(3)

g n

*Electronics* **2022**, *11*, x FOR PEER REVIEW 4 of 22

( ) is the cost function of the current position,

bile robot from the starting point to the current position and

For the cost function, the more commonly used metric is the Euclidean or Manhattan distance. The absolute value of the difference between the x-coordinates of two points and the sum of the absolute values of the differences between the y-coordinates of two points is called the Manhattan distance [16]. In this paper, the Euclidean distance is used, i.e., where ( ) , n n X Y is the position of the current point, ( ) , s s X Y is the position of the starting 133 point and( ) , t t X Y is the position of the target point. The closer the value of the function 134 hn( ) is to the actual value, the more efficient and accurate the search will be. 135

$$g(n) = \sqrt{(X\_{\text{fl}} - X\_{\text{s}})^2 + (Y\_{\text{fl}} - Y\_{\text{s}})^2} \tag{2}$$

g ( ) n

of the mobile robot from the current position to the position of the target point [15]. 128

distance. The absolute value of the difference between the x-coordinates of two points and 130 the sum of the absolute values of the differences between the y-coordinates of two points 131 is called the Manhattan distance [16]. In this paper, the Euclidean distance is used, i.e., 132

() ( ) ( ) = − +− 2 2

 X

> s

 ns

 YY

n

 X

For the cost function, the more commonly used metric is the Euclidean or Manhattan 129

h n

is the actual cost of the mo- 126

(2)

( ) is the estimated cost 127

153

$$h(n) = \sqrt{(X\_l - X\_n)^2 + (Y\_l - Y\_n)^2} \tag{3}$$

The algorithm steps are as shown in Figure 3: 155

where (*Xn*,*Yn*) is the position of the current point, (*Xs* ,*Ys*) is the position of the starting point and (*Xt* ,*Yt*) is the position of the target point. The closer the value of the function *h*(*n*) is to the actual value, the more efficient and accurate the search will be. ture of the A-star algorithm is improved so that when an obstacle is encountered during 138 the pathfinding process, instead of returning to the vicinity of the starting point for a new 139 pathfinding instance, the algorithm defines the node that the mobile robot is currently on 140

#### *2.2. Optimization of the A-Star Algorithm* as the parent node. The open list is used to store the data of the parent node and the neigh- 141