**4. Single-color femtosecond laser ablation of PMMA**

### **4.1. Introduction**

Ultrafast laser-induced ablation or breakdown of wide band gap materials, such as polymers [1, 2, 36–41], fused silica [6], and silicon [7, 42] have already been intensively studied. Among them, various kinds of polymers, such as polymethylmethacrylate (PMMA) [2, 36, 38–41], polyimide (PI) [1], polyethylene (PE) [37], polypropylene (PP), and polycarbonate (PC) [2], have drawn a lot of attention due to their potential industrial applications. Compared to nslaser ablation, the energy ablation threshold fluence of fs-laser at approximately the same incident wavelength is known to be reduced [43]. This can be attributed to the fact that the breakdown intensities in the fs regime approach that of the threshold of multiphoton ionization of which the electron densities is high enough to cause damage [35]. On the other hand, because the induced energy absorbed by electrons is much faster than that transferred to a lattice [35]; therefore, the nonthermal ablation nature of such behavior achieved by applying fs-lasers could lead to a significant reduction of heat-affected zones. Also of interest is the possibility of decreasing the threshold for ablation. For example, Stuart et al. observed a continuously decreasing threshold with a gradual transition from the thermal regime where the longer pulses (>100 ps) dominated the ablation compared with the shorter pulses (<10 ps), which is caused by multiphoton ionization and plasma formation [17].

To date, studies of single-color femtosecond laser ablation of PMMA were overwhelmingly conducted using the Ti: sapphire laser system of which the central wavelength is around 800 nm [44–46]. On the other hand, photoablation of materials with ultraviolet (UV) lasers has also gained in popularity [36, 47–49]. The mechanism for ablation of materials by UV light is mainly through the photochemical process by one-photon absorption. Most of the dielectrics, such as glass and polymer, have relatively high absorption coefficient in the UV region. This is in contrast to the commonly accepted mechanism for ablation of PMMA using 800 nm laser pulses, such as photothermal, photophysical, or multiphoton ionization and tunneling ionization, as mentioned previously. Therefore, it is of interest to conduct a comparative study of single-color femtosecond laser ablation of PMMA using the Ti: sapphire laser and its second harmonic.

the same beam path to generate 2*ω* pulses at 400 nm (*λ*2). Both beams were reflected from the silicon wafer at some incident angle, taking advantage of the fact that reflectivity of silicon varies with wavelengths and polarizations of the fundamental and second-harmonic beams (see the inset in **Figure 7**). We can control the intensity ratio *P*2*ω*/*Pω* of the two (collinear) beams by adjusting the incident angle. A pair of wedge prisms with controllable optical path difference was used to precisely adjust the relative time delay between the *ω* and 2*ω* pulses. We also employed a 5-mm-thick β-BBO to compensate the group velocity mismatch (GVM) of the two colors in the beam path. Besides, the *ω* and 2*ω* fields with original polarizations perpendicular to each other, were passed through a dual-color zero-order wave plate serving as a half-wave plate for 800 nm to make polarizations of the two colors parallel. Finally, the *ω* and 2*ω* pulses, overlapped in time with the same linear polarizations were focused on the sample. The spatial and temporal overlap and adjustment of the phase difference between the

**Figure 7.** Experimental setup for laser ablation of PMMA by femtosecond dual-colour synthesized waveforms. The polarizations of fundamental and second harmonic pulses were controlled by the half-wave plate. ND: neutral density filter; BBO: Barium borate; GVD: group velocity dispersion. The inset shows the reflectivity of the silicon wafer as a

346 Applications of Laser Ablation - Thin Film Deposition, Nanomaterial Synthesis and Surface Modification

Ultrafast laser-induced ablation or breakdown of wide band gap materials, such as polymers [1, 2, 36–41], fused silica [6], and silicon [7, 42] have already been intensively studied. Among

*ω* and 2*ω* fields were conducted using a procedure described previously [14].

**4. Single-color femtosecond laser ablation of PMMA**

function of the incident angle for both polarizations.

**4.1. Introduction**

**Figure 8.** Images of single-color (800 nm) and single-shot ablated holes. The input laser fluence are equal to (a) 2.63 J/cm2 and (b) 5.90 J/cm2 , respectively.

### **4.2. Single-shot single-color (800 nm) femtosecond laser ablation of PMMA**

In **Figure 8**, we show images of single-shot ablated holes in PMMA irradiated with femtosecond pulses at the wavelength of 800 nm. By changing the input laser fluence from 2.63 to 5.90 J/cm2 , areas of the holes are found to be equal to 155.25 and 1359.50 μm2 , respectively.

The photon energy for 800 nm is equal to 1.55 eV and the material band gap of PMMA is 4.58 eV. Therefore, more than three incident photons are needed for photoabsorption, leading to ablation. For such studies, one of the key parameter for studying the mechanism of ablation is its threshold. The method we used to define the ablation threshold value is measuring the ablated hole areas by using an optical microscope. In **Figure 9**, we have plotted hole-area of the ablated holes as a function of the irradiating laser fluence.

**Figure 9.** Hole-areas of the single-shot, single-color (800 nm) femtosecond laser ablated holes are plotted as a function of the irradiating laser fluence. Error bars are indicated.

Assuming the irradiating beam has a Gaussian spatial profile, the generally accepted scaling law for ablated holes for incident laser fluence is given by

$$\Delta D^2 = 2\nu^2 \ln\left(\frac{F}{F\_{\mu}}\right) \tag{18}$$

where *D* is the diameter of the ablated region, *w* is effective laser beam width, *F* is the incident laser fluence and *F*th denotes the ablation threshold (unit here is J/cm2 ). Following Eq. (18), the ablation threshold *F*th can be determined by fitting the experimental data to be 2.63 J/cm2 .

### **4.3. Single-shot single-color (400 nm) femtosecond laser ablation of PMMA**

To compare, we conducted similar ablation studies with exciting wavelength at 400 nm. Recall that the material band gap of PMMA is 4.58 eV, which means the dominated mechanism for photoablation on PMMA at 400 nm is also multiphoton absorption. The photon energy for 400 nm is equal to 3.1 eV. Therefore, more than two incident photons are needed for photoabsorption, leading to ablation. In **Figure 10**, we show images of single-shot ablated holes in PMMA irradiated with femtosecond pulses at the wavelength of 400 nm. By changing the input laser fluence from 1.78 to 3.92 J/cm2 , the hole areas are found to increase from 155.25 to 1359.50 μm2 , respectively.

**Figure 10.** Images of single-color (400 nm) and single-shot ablated holes. The input laser fluence are equal to (a) 1.78 J/cm2 and (b) 3.92 J/cm2 , respectively.

**Figure 11** shows determination of the ablation threshold in the case of exciting wavelength at 400 nm. It can be seen that the same scaling behavior is observed in the case of ablation by the near IR beam. Our data show that the ablation threshold for PMMA irradiated by the near UV light of 400 nm is about 1.38 J/cm2 .

**Figure 11.** Single-colour ablation results for PMMA irradiated by femtosecond laser pulses with a central wavelength 400 nm. The fitted ablation threshold *F*th is equal to 1. 38 J/cm2 .


**3.1.2 Genetic parameters**

404 Fuzzy Inference System – Theory and Applications

These algorithms are computationally very simple, are quite powerful. In addition, they are not limited by assumptions about the search space, for continuity, existence of derivatives,

The basic principle of genetic operators is to transform the population through successive generations, extending the search until you reach a satisfactory outcome. Genetic operators are needed to enable the population to diversify and keep adaptation characteristics

The operator mutation is necessary for the introduction and maintenance of genetic diversity of the population, arbitrarily changing one or more components of a structure chosen, as is illustrated in Figure 7, thus providing the means for introduction of new elements in the population. Thus, the mutation ensures that the probability of reaching any point in the search space will never be zero, in addition to circumvent the problem of local minima, because with this mechanism, slightly changes the search direction. The mutation operator is applied to individuals with a probability given by the mutation rate *Pm*; usually

The crossing is the operator responsible for the recombination of traits of parents during play, allowing future generations to inherit these traits. It is considered the predominant genetic operator, so it is applied with probability given by the crossover rate *Pc*, which must

a. One-point: a crossover point is chosen and from this point the parental genetic information will be exchanged. The information prior to this point in one of the parents is related to information subsequent to this point in the other parent, as shown in the

Fig. 8. Example of crossover from one-point: (a) two individuals are chosen; (b) a crossover point (2) is chosen; (c) the characteristics are recombined, generating two new individuals

b. Multi-points: is a generalization of this idea of an exchange of genetic material through

c. Uniform: don't use crossing points, but determines, through a global parameter, which

points, where many crossing points can be used.

the probability of each variable be exchanged between parents.

uses a small mutation rate, because it is a genetic operator secondary.

This operator can also be used in several ways; the most commonly used are:

and so on.

**3.1.1 Genetic operators**

acquired by previous generations.

Fig. 7. Example of mutation

be greater than the rate of mutation.

example in Figure 8.

It is also important to analyze how some parameters influence the behavior of genetic algorithms in order to establish them as the needs of the problem and available resources.

**Population size.** The size of the population affects the overall performance and efficiency of AGs. With a small population performance may fall, because this way the population provides a small coverage of the search space of the problem. A large population typically provides a representative coverage of the problem domain, and preventing premature convergence solutions to local rather than global. However, for working with large populations, larger computational resources are required, or that the algorithm works by a much longer time period.

**Passing Rate.** The higher this ratio, the faster new structures will be introduced in the population. But if this is too high, the majority of the population will be replaced and can be lost high fitness structures. With a low value, the algorithm can become very slow.

**Mutation rate.** A low mutation rate prevents a given position stay stagnant in a value, and allow to reach anywhere in the search space. With a very high search becomes essentially random.

## **3.2 Particle Swarm Optimization**

The optimization method called Particle Swarm Optimization (PSO) as other meta-heuristics recently developed, simulates the behavior of systems making the analogy with social behaviors. PSO was originally inspired by biological partner behavior associated with group of birds (Goldberg, 1989). This topic will be discussed in more detail after the basic algorithm is described.

The PSO was first proposed by John Kennedy and Russell Eberhart (1995a, 1995b). Some of the interesting features of PSO include ease of implementation and the fact that no gradient information is required. It can be used to solve a range of different optimization problems, including most of the problems can be solved through genetic algorithms; one can cite as an example some of the applications, such as neural network training (Lee & El-Sharkawi, 2008) and to minimize various types of functions (Eberhart et al., 1996).

Many popular optimizations algorithms are deterministic, as the gradient-based algorithms. The PSO, like its similar, belonging to the family of Evolutionary Algorithm is an algorithm of stochastic type that needs no gradient information derived from error function. This allows the use of PSO in functions where the gradient is unavailable or the production of which is associated with a high computational cost.


Furthermore, 68 facial landmarks were imaged and entered into the network to extract facial landmark features, and an experiment was conducted to determine the best method for processing and inputting these facial landmark images. As the results of the face detector and facial landmark detector illustrated a jittering pattern, we examined the extent to which the stress recognition performance was affected when this phenomenon was prevented by applying min–max normalization and Gaussian blurring to the facial landmark images. The experimental results are listed in Table 7.

**Table 7.** Comparison of facial landmark feature extraction methods.

Att: spatial and temporal attention modules, Norm: min–max normalization, Blur: Gaussian blurring.

The experimental results demonstrated that the performance decreased when only the landmark image was used or only min–max normalization was applied. However, when min–max normalization and Gaussian blurring were both applied to the landmark images, the performance increased. Thus, when both min–max normalization and Gaussian blurring were used, the jittering phenomenon was prevented.

### 5.3.4. Clip Length and Number of Frames

Finally, we analyzed the impact of the proposed method on the performance by varying the clip length and number of frames. First, we experimented by changing the clip length, which is a unit used in training and testing, to 1 s, 2 s, 5 s, 10 s, and 30 s; the results are listed in Table 8. To match the experimental conditions as much as possible, we used 24 frames for 1 s, and 48 frames were used in the remaining experiments.

**Table 8.** Effect of clip length on the performance.

The experimental results demonstrated that the best performance occurred when the clip length was 2 s. It was possible to identify the cues that indicated stress in 2 s clips, and the temporal change was learned well using 48 consecutive frames. In contrast, we randomly selected 48 frames for clips longer than 2 s and used them for training and testing; hence, the discontinuity between frames could have an adverse effect on learning the temporal change. Next, we experimented by changing the number of frames constituting one clip to 8, 16, 32, 48, and 64, and the results were the same as in Table 9. To match the experimental conditions as much as possible, we used 2.7 s clips for 64 frames, while the other experiments used 2 s clips.

**Table 9.** Effect on the performance of the number of frames.

The experimental results demonstrated that the highest performance was achieved when 48 frames were used. This setting exhibited the highest performance when all 48 frames of the 2 s clips were used because it is necessary to find the overall spatial and temporal facial changes when recognizing stress. In contrast, when the clip length exceeded 2 s, recognition was hampered by the increased amount of unnecessary information, as in the above experiment.

### *5.4. Comparison with Other Methods*

We evaluated the stress recognition performance of the proposed method, as well as various other methods. We compared the proposed method with widely used deeplearning networks that have demonstrated high performance [46,47,57,64–67]. The HOG– SVM method, which combines the widely used handcrafted features, HOG [68], and the classical classifier SVM [69], was used for comparison. In addition, current deeplearning-based recognition methods [41–43,45] using spatial–temporal facial information were also used for performance comparison. These methods were used because an emotion recognition network could be considered similar to a stress recognition network.

The experimental results of the proposed method and other methods are listed in Table 10, along with each method's feature dimension. In general, a higher feature dimension indicates a higher discriminating power, but because the computational complexity increases, lower feature dimensions that exhibit high performance are preferable.

As depicted in the experimental results, the proposed method had the highest accuracy, 66.8409%, even though features with a relatively low dimensional number of 768 were used. Even when the facial landmark feature was not used, it exhibited an accuracy of 65.3396% with a small 512-dimensional feature. SE-ResNet-18 had the highest performance, at 65.7013%, among the widely used deep-learning networks. This network uses attention modules, which seems to have a positive effect on the stress recognition performance.

By contrast, VGG-16 and ResNet-50 exhibited low performance despite using a relatively high number of feature dimensions, i.e., 2048. This result demonstrates that these methods have a network structure that is unsuitable for stress recognition. The HOG–SVM method used a relatively high number of feature dimensions, i.e., 1764, but exhibited the lowest performance, i.e., 50.9153%. Thus, it was demonstrated that the discriminating power of the handcrafted features was lower than that of the deep-learning networks.

Examining the results of methods using spatial–temporal facial information, the method using the 2D-CNN, LSTM, and facial landmarks demonstrated low performance, i.e., 58.3432%. This result indicates that the facial landmark information was not utilized satisfactorily because the coordinates of the facial landmarks were simply input into the network. Furthermore, the method using the 3D-CNN with hyperparameter optimization

exhibited high performance at 65.9372%. Thus, even a simple network can exhibit high performance through appropriate hyperparameter optimization.

We also compared the performance with the method using a physiological signal database [13]. It can be seen that the performance of that method was higher than ours at 74.1%. However, unlike our method, which classified three stress states, this method classified two stress states. In addition, since this method uses physiological signal data, a direct comparison with our method is not possible. Therefore, our approach, which showed the highest performance when there were three stress states, was quite competitive as it offered finer distinctions. Furthermore, as mentioned before, our method does not require biosensors and has the advantage of being able to be used for more diverse applications using images.

Furthermore, we compared the performance with the previous video-based stressrecognition method [30]. The performance of the method was high at 64.6481%, but the performance was lower than that of our proposed method. Therefore, the experimental results in Table 10 show that the performance of the proposed method was higher than that of the other methods. These results indicate that the proposed method is superior to other methods in detecting the overall spatial and temporal changes of the face.

We present the sensitivity and specificity rates along with classification accuracy in Table 10. It can be confirmed that the proposed method showed the best performance in both sensitivity and specificity, as well as accuracy.

We also output the feature maps and attention map obtained from the spatial attention module, and the results are shown in Figure 7. In the case of the attention map, it can be seen that a higher weight was assigned to the lower part of the face. However, in the case of the feature map, it can be seen that it is difficult to identify which features have been learned because the resolution was as low as 4 × 4. Therefore, we drew a picture of the Grad-cam [70], which shows which part of the image was mainly viewed and determined the prediction. We drew the Grad-cam results for the facial image, as well as the facial landmark image, and the results are shown in Figure 7. As can be seen from Figure 7, the network predicted the stress level primarily by considering the areas around the eyes and mouth.

**Figure 7.** Feature maps, attention map, and Grad-cam images output from an example facial image.

In addition, temporal attention weights were visualized to check whether temporal attention was well applied, and the result is as shown in Figure 8. In the neutral state, the change rate of the weight was not large; however, in the stressed state, the change rate was large. It can be seen that the weight was higher for images in which the change in facial expression was large. This showed that the temporal attention module was working properly.

The classification accuracy for each of the proposed method's classes is listed in Table 11. When the facial landmark feature was used, the proposed method demonstrated higher performance for all three classes than when it was not used. This result implies that the facial landmark feature effectively complements the facial image feature. However, even if the facial landmark feature is used in the proposed method, its classification of the neutral state was superior to its classification of the stress states. Thus, it is challenging to find overall spatial and temporal facial changes that appear when people are under stress. Especially under low stress, the changes are smaller, so they are more difficult to pinpoint. We also output the confusion matrix of the proposed method without and with the facial landmark feature, and the results are shown in Figure 9. Figure 9 shows that the overall

performance improved when the facial landmark feature was used compared with not using it.

**Figure 8.** Visualization of the temporal attention weight in three stress states. The higher the height of the bar on the image, the greater the weight is.

**Table 11.** The proposed method's classification accuracy for each stress state with and without the facial landmark feature.

**Figure 9.** Confusion matrix of the proposed method (**a**) without and (**b**) with the facial landmark feature.

We plotted a histogram of the accuracy of each subject in the proposed method, as shown in Figure 10. The histogram shows how the average performance of the three classes is distributed for all subjects. More specifically, five subjects with an accuracy of 30%∼40% means that the number of subjects with an average performance of three classes between 30% and 40% is five. The interval with the largest number of subjects was between 60% and 70%, and the average performance of the three classes in our method from Table 11 also involved this interval.

**Figure 10.** Histogram of the accuracy for each subject in the proposed method.

To evaluate the performance of the video unit, we performed classification by dividing all 5 min and 10 min videos into 2 s clips. For each subject, there were two 5 min videos for the neutral class and one 10 min video for the low- and high-stress classes. If the ratio of correctly classified clips was greater than the threshold, the video was counted as correctly classified and the accuracy was measured. The video unit performance of the proposed method is shown in Table 12, and it is possible to grasp the trend of the performance change according to the threshold change. Since the accuracy was calculated using the results of Table 10 learned by the cross-validation method, the cross-validation method was also applied to these results. If the threshold was set to 50%, the video unit performance was better than the 2 s clip unit performance. For the three classes, the threshold value of 50% can be seen as a reasonable value.

**Table 12.** Video-based stress recognition accuracy in the proposed method obtained by changing the threshold.

### **6. Conclusions**

In this paper, a stress-recognition method using spatial–temporal facial information was proposed using deep learning. To use deep learning technology, we built and released a large image database for stress recognition. In the proposed method, we used a spatial attention module that assigns a high weight to the stress-related regions of the facial image. Using a temporal attention module that assigns a high weight to frames that are highly related to stress from among several frames in the video, we improved the feature's discriminating power. Furthermore, using features extracted from the facial landmark information, we supplemented the discriminating power of the feature extracted from the facial image.

We designed the loss function so that the network learning proceeds effectively, considering the characteristics of the constructed database. We evaluated the proposed method on our constructed database, and it exhibited higher performance than existing deep-learning-based recognition methods. However, our approach has a limitation in that it would find it difficult to recognize stress in people who do not display much change in their facial expressions. In the future, to mitigate this limitation, a study on stress recognition based on multimodal data will be conducted using voice data, which is closely related to

stress, along with the images. In addition, research in more difficult environments such as occlusion on the face will be conducted as future work.

**Author Contributions:** T.J. developed the methodology, led the entire research including the evaluations, and wrote and revised the manuscript. H.B.B., Y.L. and S.J. designed the experiments and analyzed the results. S.L. guided the research direction and verified the research results. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was supported by Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government, Ministry of Science and ICT (MSIT) (No.2016-0-00197, Development of the high-precision natural 3D view generation technology using smart-car multi sensors and deep learning).

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Institutional Review Board of Yonsei University (date of approval: 26 September 2019).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Data Availability Statement:** The data presented in this study are openly available on IEEE DataPort at https://dx.doi.org/10.21227/17r7-db23 (accessed on 8 November 2021).

**Acknowledgments:** We would like to express our gratitude to the Korea government, Ministry of Science and ICT (MSIT) and Yonsei University Office of Research Affairs.

**Conflicts of Interest:** The authors declare no conflict of interest.

### **Appendix A**

**Table A1.** Summary of the database construction settings and database contents.

### **References**

### *Article* **Restoration of Motion Blurred Image by Modified DeblurGAN for Enhancing the Accuracies of Finger-Vein Recognition**

**Jiho Choi, Jin Seong Hong, Muhammad Owais, Seung Gu Kim and Kang Ryoung Park \***

Division of Electronics and Electrical Engineering, Dongguk University, 30 Pildong-ro 1-gil, Jung-gu, Seoul 04620, Korea<EMAIL_ADDRESS>(J.C<EMAIL_ADDRESS>(J.S.H<EMAIL_ADDRESS>(M.O<EMAIL_ADDRESS>(S.G.K.)

**\*** Correspondence<EMAIL_ADDRESS>
**Abstract:** Among many available biometrics identification methods, finger-vein recognition has an advantage that is difficult to counterfeit, as finger veins are located under the skin, and high user convenience as a non-invasive image capturing device is used for recognition. However, blurring can occur when acquiring finger-vein images, and such blur can be mainly categorized into three types. First, skin scattering blur due to light scattering in the skin layer; second, optical blur occurs due to lens focus mismatching; and third, motion blur exists due to finger movements. Blurred images generated in these kinds of blur can significantly reduce finger-vein recognition performance. Therefore, restoration of blurred finger-vein images is necessary. Most of the previous studies have addressed the restoration method of skin scattering blurred images and some of the studies have addressed the restoration method of optically blurred images. However, there has been no research on restoration methods of motion blurred finger-vein images that can occur in actual environments. To address this problem, this study proposes a new method for improving the finger-vein recognition performance by restoring motion blurred finger-vein images using a modified deblur generative adversarial network (modified DeblurGAN). Based on an experiment conducted using two open databases, the Shandong University homologous multi-modal traits (SDUMLA-HMT) finger-vein database and Hong Kong Polytechnic University finger-image database version 1, the proposed method demonstrates outstanding performance that is better than those obtained using state-of-theart methods.

**Keywords:** Finger-vein recognition; motion blur image restoration; modified DeblurGAN; CNN

### **1. Introduction**

There are several types of measurable human biometrics, including those of voice, face, iris, fingerprint, palm print, and finger-vein recognition. Among these, finger-vein recognition has the following advantages, (1) finger-vein patterns are hidden under the skin. Therefore, they are generally invisible, making them difficult to forge or steal. (2) Noninvasive image capture ensures both convenience and cleanliness and is more suitable for a user. (3) As most people have ten fingers, if an unexpected accident occurs with one finger, the other finger can be used for authentication [1]. However, due to various factors such as light scattering in the skin layer caused by near-infrared (NIR) light, focus mismatch of a camera lens, differences in finger thickness, differences in depth between the surface of the skin and vein, and finger movements, blurring may occur when capturing finger-vein images. Blurred images generated in these kinds of blur can significantly reduce finger-vein recognition performance. Therefore, image restoration through a deblurring method is necessary. Extensive research has been conducted for restoring skin scattering blur that occurs frequently [2–9], and several studies have been conducted on optical blur caused by the difference in the distance from a camera lens to the finger vein and finger thickness [10,11]. Motion blur can occur frequently, due to finger movement. However, no study has been conducted for motion blurred finger-vein image restoration.

**Citation:** Choi, J.; Hong, J.S.; Owais, M.; Kim, S.G.; Park, K.R. Restoration of Motion Blurred Image by Modified DeblurGAN for Enhancing the Accuracies of Finger-Vein Recognition. *Sensors* **2021**, *21*, 4635. https://doi.org/10.3390/s21144635

Academic Editor: Loris Nanni

Received: 10 May 2021 Accepted: 3 July 2021 Published: 6 July 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

Although, during finger-vein image capture, a finger is fixed to the image capturing device to some extent; however, Parkinson's disease, physiologic tremors, dystonia, and excessive stress may cause hand tremors. Due to these reasons, motion blur can occur. Furthermore, with the recent expansion of non-contact devices due to COVID-19, motion blur occurs more in the input image, and the resulting motion blurred image causes severe performance degradation during finger-vein recognition. To solve this problem, the restoration of a motion blurred finger-vein image is essential.

Conventional image restoration methods can be categorized into blind and non-blind deblurring [12]. Early non-blind deblurring methods perform deblurring, assuming that blur kernels are known. Blur kernel is deduced from knowledge of the image formation process (e.g., amount of motion or defocus blur and camera sensor optics), calculated from the test image, or measured through point spread function (PSF) [13]. Using these methods, the original sharp image can be obtained through deconvolution by estimating the blur kernel. However, when a non-blind restoration method is applied, the recognition performance can be reduced if images are acquired from various devices and show difference blurring characteristics in the spatial domain. Moreover, there are limitations to applying non-blinded methods to each case because various types of distortions occur when capturing an image in actual environments. Also, most blur kernels are unknown in actual environments, and it is time-consuming to estimate blur kernels.

Contrary to the non-blind deblurring method, the blind deblurring methods proceed with deblurring, assuming that blur kernels are unknown [12,14,15]. A generative adversarial network (GAN) that combines the blind deblurring method and the training-based method has also been studied to solve the problems arising from non-blinded deblurring [12,14]. GAN is a network that generates an image by finding an optimal filter using weights trained from the training data. Therefore, using GAN has the advantage of being robust even if images have various distortions. Also, there is no need to estimate the blur kernel directly, and restoration can be performed through training. Considering these reasons, we propose a method of performing motion blurred finger-vein image restoration using the newly proposed modified DeblurGAN and a method of performing restored finger-vein image recognition using deep CNN. The main contributions of our paper are as follows:

This paper is organized as follows: Section 2 provides an overview of the previous studies, and the proposed method is explained in Section 3. In Section 4, comparative experiments and experimental results with analysis are described. Finally, in Section 5, the conclusions of this paper are explained.

### **2. Related Works**

Previous studies on blurred finger-vein image restoration have been conducted on the restoration of skin scattering or optical blur, and studies related to motion blur restoration have not been conducted. Therefore, previous studies were analyzed in terms of finger-vein recognition without blur restoration, with skin scattering blur restoration, and with optical blur restoration. Such methods can be further categorized into handcrafted feature-based and deep-feature-based finger-vein recognition for analysis.

### *2.1. Finger-Vein Recognition without Blur Restoration*

For the handcrafted feature-based finger-vein recognition without blur restoration method, Lee et al. [17] proposed a method for finger-vein recognition by aligning the image using minutia points extracted from the finger-vein region, extracting finger-vein features using a local binary pattern (LBP), and calculating the Hamming distance using the extracted features. Peng et al. [18] applied Gabor filters having eight orientations to the original finger-vein image and extracted the finger-vein pattern by the fusion of the image with the vein pattern highlighted. They proposed a scale-invariant feature transform (SIFT) feature matching method based on the extracted finger-vein patterns. The method proposed in the study of [18] has the advantage that recognition performance is improved when an optimal filter is accurately modeled. However, this method can cause performance degradation when the filter is applied to finger-vein images having multiple characteristics, and since this experiment was conducted in a constraint environment, it is not robust to image variants, such as illumination or misalignment. Moreover, they did not consider the blur that could occur when capturing a finger-vein image.

Deep feature-based methods have been studied to overcome the drawbacks of these handcrafted feature-based methods. Although a deep-learning-based method was not used, Wu et al. [19] performed dimension reduction and feature extraction of a finger-vein image using a principal component analysis (PCA) and a linear discriminant analysis (LDA). They proposed a finger-vein pattern identification method based on a support vector machine (SVM), which used the PCA- and LDA-extracted features. Hong et al. [20] and Kim et al. [21] proposed finger-vein verification methods to distinguish genuine (authentic) matching (matching images of the same class), and imposter matching (matching images of different classes) using the difference image of enrolled and input images as input to a CNN. Qin et al. [22] created vein-pattern maps, calculated the finger-vein feature probability for each pixel, and labeled veins and backgrounds. Subsequently, training was conducted by dividing the original image into an N×N size, and the probability that the final input image was the vein pattern was calculated. Song et al. [23] and Noh et al. [24,25] proposed a shift-matching finger-vein recognition method using a composite image. Qin et al. [26] proposed a fingervein verification method that combined a CNN and long short-term memory (LSTM). They assigned labels through handcrafted finger-vein image segmentation techniques and extracted finger-vein features using stacked convolutional neural networks and long short-term memory (SCNN-LSTM). Genuine and imposter matching were verified using feature matching between supervised feature encoding and enrollment databases using extracted features. These studies on deep feature-based finger-vein recognition have a limitation that an intensive training process is required, and there is a disadvantage that they did not consider blur that can occur when capturing finger-vein images.

### *2.2. Finger-Vein Recognition with Skin Scattering Blur Restoration*

Lee et al. [2] proposed a method for restoring skin scattering blur by measuring a PSF of a skin scattering blur and using a constrained least squares (CLS) filter. Yang et al. [3,4] performed scattering-removal by calculating light-scattering components of a biological optical model (BOM). Yang et al. [5] performed scattering effects removal from fingervein images by considering an anisotropic diffusion, and gamma correction (ADAGC), weighted biological optical model (WBOM), Gabor wavelet, non-scattered transmission map (NSTM), and inter-scale multiplication operation. Shi et al. [6] used haze-removal techniques based on Koschmieder's law to remove scattering effects in finger-vein images. Yang et al. [7] used multilayered PSF and BOM to restore blurred images. Furthermore, Yang et al. [8] proposed a scattering-effect removal method using a BOM-based algorithm that measured the scattering component with the transmission map. You et al. [9] designed a bilayer diffusion model to simulate light scattering and measured the parameters of a

bilayer diffusion model through blur-Steins unbiased risk estimate (blur-SURE). Image restoration methods were also proposed based on these parameters with the multi-Wiener linear expansion thresholds (SURE-LET). However, these studies have the disadvantages that scattering blur parameters must be accurately estimated, and parameters must be re-estimated when the domain between the image used for estimation and the test image is different.

### *2.3. Finger-Vein Recognition with Optical Blur Restoration*

Lee et al. [10] proposed a blurred finger-vein image restoration method that considers both optical and scattering blur using PSF and CLS filters. They restored blurred fingervein images by considering both optical blur components and scattering blur components and improved recognition performance. However, this method requires that parameters should be accurately predicted when measuring two PSFs to improve performance, causing extensive processing time. Choi et al. [11] proposed a finger-vein recognition method by restoring the optical blur included in the original finger-vein image based on modified conditional GAN. This method has the advantage that it can be applied to images acquired from various environments but has the disadvantage that it does not consider more complex motion blur that can occur during image acquisition.

As such, most of the previous studies did not focus on motion blur that can occur from the movement of fingers in finger-vein recognition and did not consider the image restoration associated with the motion blur. Therefore, we propose a new method of restoring a motion blurred finger-vein image using the modified DeblurGAN and recognizing the restored image using a deep CNN.

Point spread functions (PSFs) for skin scattering and optically blurred images are completely different from that for motion blurred images [2–11,27,28]. Therefore, the methods developed for skin scattering or optically blurred images cannot be used directly to solve the motion blurring issue. In the case of the handcrafted feature-based method of Table 1, the PSFs for skin scattering or optically blurred images should be replaced by the PSF for motion blurred images with optimal parameters of PSF. In case of deep featurebased method of Table 1, the CNN and GAN models for skin scattering or optically blurred images should be retrained with motion blurred images in addition to the modification of layers or filters of CNN and GAN models.

**Table 1.** Comparisons of the previous and proposed finger-vein image restoration methods.

**Table 1.** *Cont.*

Table 1 presents a comparison of the advantages and disadvantages of the proposed method and the previous studies.

### **3. Proposed Method**

### *3.1. Overview of the Proposed Method*

Figure 1 shows the overall flowchart of the proposed method. After acquiring finger images (step (1)), the finger region of interest (ROI) is detected using preprocessing method (step (2)). Then, the motion blurred finger-vein image is restored using the proposed modified DeblurGAN (step (3)). One difference image is then generated from the restored enrolled and recognized images (step (4)). Lastly, based on the output score obtained by inputting the difference image in the deep CNN, finger-vein recognition is performed to distinguish genuine (authentic) or imposter matching (step (5)).

**Figure 1.** Flowchart of the proposed method.

### *3.2. Preprocessing the Finger-Vein Image*

The first part of preprocessing removes unnecessary background regions and finds the finger-vein ROI. The captured image is then binarized to obtain the image shown in Figure 2b. However, even if binarization is performed, the background is not completely removed, so an edge map is created using a Sobel filter. A difference image is then generated using the created edge map and the binarized image. By applying the area threshold method [29] to the generated difference image, an image with the background removed as shown in Figure 2c is obtained. Then, in order to correct misalignment caused by in-plane rotation of the finger image, which degrades recognition performance, secondorder moments of the binarized mask *R* (Figure 2c), are calculated using Equation (1).

**Figure 2.** Example of background removal and in-plane rotation compensation: (**a**) original image; (**b**) binarized image; (**c**) background removed image; (**d**) in-plane rotation compensation.

Note that *f*(*x*, *y*) and (*mx*, *my*) represent image pixel values and central coordinates, respectively. Based on these values, the rotation angle *θ*, in Equation (2) is calculated to compensate for the in-plane rotation [30]. The compensated image, shown in Figure 2d, is obtained from this process.

$$\begin{array}{c} a\_{11} = \frac{\sum\_{(x,y)\in R} \left(y - m\_y\right)^2 \cdot f(x,y)}{\sum\_{(x,y)\in R} \prod(x,y)}\\ a\_{12} = \frac{\sum\_{(x,y)\in M} (x - m\_x) \left(y - m\_y\right) \cdot f(x,y)}{\sum\_{(x,y)\in R} \prod(x,y)}\\ a\_{22} = \frac{\sum\_{(x,y)\in R} (x - m\_x)^2 \cdot f(x,y)}{\sum\_{(x,y)\in R} \prod(x,y)} \end{array} \tag{1}$$

$$\theta = \begin{cases} \tan^{-1} \left\{ \frac{a\_{11} - a\_{22} + \sqrt{(a\_{11} - a\_{22})^2 + 4a\_{12}^2}}{-2a\_{12}} \right\} \text{if } a\_{11} > a\_{22} \\\\ \tan^{-1} \left\{ \frac{-2a\_{12}}{a\_{22} - a\_{11} + \sqrt{(a\_{22} - a\_{11})^2 + 4a\_{12}^2}} \right\} \text{if } a\_{11} \le a\_{22} \end{cases} \tag{2}$$

As shown in Figure 3a, the left and right ends of the finger are the regions of the thick area or region with a fingernail where NIR lighting is not well-transmitted. Thus, these regions are inappropriate for recognition because vein patterns are not likely to be captured accurately. Therefore, the image, shown in Figure 3c, is obtained by removing the left and right sides by a predetermined size to which in-plane rotation compensation is applied. By performing erosion operation, component labeling process, and dilation operation [27], the unnecessary region for finger-vein recognition such as the upper right corner of Figure 3c, is removed. As a result of this process, an image as shown in Figure 3d is created. Since the vein pattern is not acquired by bright illumination, the black area of the finger area is not required for recognition. An ROI mask is obtained by using a 4 × 20 mask to fill the black area with the average pixel values around it (Figure 3e). In details, as shown in the red-dashed circles of the lower boundary of finger in Figure 3a, there exists bright pixels inside of finger caused by excessive illumination, which causes the error of binarization of lower boundary as shown in Figure 3b–d. Therefore, we applied 4 × 20 mask to the binarized image of Figure 3d. At each convolution position of mask, the average pixel value within 4 × 20 area (except for the black pixels of Figure 3d) is assigned to the binarized image of Figure 3d. That is, if the majority pixels within 4 × 20 area is white (255), white pixel is assigned. Then, the inaccurate black pixels of the red-dashed

circles of Figure 3d are replaced by the white pixels of finger region as shown in the lower boundary of Figure 3e.

**Figure 3.** Extracting finger-vein ROI: (**a**) original finger image, (**b**) in-plane rotation compensation, (**c**) left and right areas removal, (**d**) component labeling, (**e**) ROI mask after filling black area of finger region, and (**f**) obtained ROI image.

The mask size (4 × 20) generalizes on the images of different resolutions. To confirm this, we used two open databases of the Shandong University homologous multi-modal traits (SDUMLA-HMT) finger-vein database [31] and the Hong Kong Polytechnic University finger-image database version 1 [29] in our research.

### *3.3. Modified DeblurGAN-Based Finger-Vein Image Restoration*

The principal objective of enhancement is to process the image so that the result is more suitable than the original image for a specific application [27]. Therefore, although image enhancement is mostly a subjective process, while image restoration is a generally objective process. Because image restoration is an attempt to reconstruct a degraded image using prior knowledge of degradation, the restoration method must focus on applying degradation modeling to restore the original image and the inverse process. The blur model based on the above process can be expressed as follows [28]:

$$\log(\mathbf{x}, \ y) = h(\mathbf{x}, \ y) \* f(\mathbf{x}, \ y) + \eta(\mathbf{x}, \ y) \tag{3}$$

Here, *g*(*x*, *y*) is a degraded (blurred) image, *h*(*x*, *y*) is a spatial representation of a degradation function (*H*), ∗ is a convolution operation, *f*(*x*, *y*) is an input image, and *η*(*x*, *y*) is an additive noise. If the above conditions are given, the goal of restoration is to obtain ˆ *f*(*x*, *y*), which is the estimation of an original image. The more accurately *h*(*x*, *y*) and *η*(*x*, *y*) are estimated, ˆ *f*(*x*, *y*) and *f*(*x*, *y*) become closer [28]. However, from *g*(*x*, *y*), which is the image obtained from various environments, it is extremely difficult to estimate *h*(*x*, *y*) and *η*(*x*, *y*) accurately. Furthermore, when images having different characteristics than those used for estimation are input, the estimated *h*(*x*, *y*) and *η*(*x*, *y*) may sometimes not be applicable. Considering these facts, this study proposes a training-based restoration model, the modified DeblurGAN, and we aim to ensure the restored finger-vein image *Fres*, becomes similar to the original finger-vein image *Fori*, through training without separately estimating *h*(*x*, *y*) and *η*(*x*, *y*) when a motion blurred finger-vein image *Gblur*, is given.

A deblurring task can be generally divided into blind and non-blind deblurring. For the non-blind deblurring method, deblurring is performed assuming that the blur kernel (*h*(*x*, *y*)) is known, whereas, for the blind deblurring method, deblurring is performed assuming that the blur kernel is not known [12]. In a general environment, a blind kernel is not known, and it is time-consuming to directly estimate it. In this study, we assume that the blur kernel is unknown, similar to the general environment. Also, it proposes a restoration method applicable for motion blurred finger-vein images obtained from various environments, so this study can be considered a blind deblurring task. Because the original DeblurGAN exhibits good performance in a blind motion-deblurring task [12], we determined that it would be effective in this study as well. Therefore, we propose a modified DeblurGAN. The generator of the modified DeblurGAN used in this study is shown in Figure 4 and Table 2, and the discriminator is shown in Figure 5 and Table 3. A more detailed explanation is provided in the next subsection.

**Figure 4.** Generator of the modified DeblurGAN.

**Table 3.** Descriptions of the discriminator in modified DeblurGAN (\* means the output image or target image of Figure 5).

### 3.3.1. Generator

A GAN generally comprises generator and discriminator models in which the adversarial training between the two gradually improves the performance of both. The generator of the original DeblurGAN has one convolution block, two strided convolution blocks with strides of 1/2, nine residual blocks (ResBlocks) [32], and two transposed convolution blocks [12]. Each ResBlock consists of a convolution layer, an instance normalization layer, and a rectified linear unit (ReLU) for activation [33]. Compared with the original DeblurGAN, the following two aspects were modified for this study.

First, a dropout [34] is removed. In the original DeblurGAN, a dropout ratio of 0.5 is applied to each residual block of the generator, and the same ratio is applied for inference. Generally, a dropout is effective as a regularization method for avoiding overfitting, but it can cause the modification of a vein pattern in the restored output image, due to the randomness of a dropout when applied to a restoration task. The modified vein pattern then has different features from the original finger-vein image, which results in degraded performance. Rather than creating a variety of outputs in which the vein pattern is deformed, the generated pattern information needs a deterministic output that is similar to the original as possible, therefore, dropout.

Second, the number of parameters is reduced by modifying the residual blocks. Large parameters can increase the inference time when applied to an actual environment, and increased inference time can cause the inefficiency of the system. The original DeblurGAN used the GoPro [35] and Kohler datasets [36] and applied nine residual blocks to the generator. In this study, the existing nine residual blocks were reduced to six to shorten the inference time by reducing the number of parameters. Also, by modifying the residual blocks as shown in Figure 6, feature information is maintained in the layer prior to the next convolution layer, and the number of parameters is reduced. The width and height of feature map are reduced by passing through convolution layer, which usually causes the reduction of important feature information [32]. Therefore, by comparing Figure 6a,b, the second 3 × 3, 256 Conv layer is removed in our modified residual block, which can maintain feature information in the layer prior to the next convolution layer. In addition, the number of parameters is reduced by removing the second 3 × 3, 256 Conv layer in the modified residual block. Consequently, the number of parameters of the generator is reduced from 6.0 to 4.2 million.

**Figure 6.** Architectures of original and modified residual blocks: (**a**) residual block in original DeblurGAN; (**b**) a modified residual block in the modified DeblurGAN.

### 3.3.2. Discriminator

The structure of the discriminator is shown in Figure 5 and Table 3. The discriminator of the modified DeblurGAN proposed in this study has the same structure as the discriminator of the original DeblurGAN, which used the Wasserstein WGAN gradient penalty (GP) [37]. For a GAN, the Nash equilibrium in a non-convex system must be found using continuous and high-dimensional parameters for smooth training, however, the existing GAN [38] cannot solve this problem, therefore, it fails to converge [39]. In the case of DeblurGAN, WGAN-GP is used as a critic function using Wasserstein distance and the gradient penalty methods proposed in [37]. Thus, a structure that is robust to generator structure selection and at the same time enables stable training is proposed. In this study, these advantages of the discriminator of the original DeblurGAN are adopted.

### 3.3.3. Loss

In the case of the original DeblurGAN, a perceptual loss is applied to perceptually hard to distinguish between the generated image and the real sharp image and to restore finer texture detail [12]. A perceptual loss refers to the difference in feature maps between the generated and target images, which can produce better results than the loss that generates blurry results by calculating the pixel-wise average difference such as L1 or L2 loss. The perceptual loss function used in this study, based on the previous study [12], can be defined as follows:

$$\mathcal{L}\_X = \frac{1}{\mathcal{W}\_{\mathcal{B}} \mathcal{H}\_{\mathcal{E}}} \sum\_{x=1}^{\mathcal{W}\_{\mathcal{E}}} \sum\_{y=1}^{H\_{\mathcal{E}}} \left( \mathcal{Q} \left( F^{ori} \right)\_{x,y} - \mathcal{Q} \left( G\_{\theta\_G} \left( F^{blur} \right) \right)\_{x,y} \right)^2 \tag{4}$$

where ∅ is the feature maps extracted from the ImageNet pretrained network. For the original DeblurGAN, the feature maps extracted from the third convolution layer before the third max-pooling layer in the visual geometry group (VGG)-19 [40] are used for perceptual loss. *W*∅ and *H*∅ are the width and height of feature maps, respectively. In a classification network, such as that of a VGG, abstracted features extracted from a higher layer preserve the overall spatial structure, whereas low-level features, such as color, corner, edge, and texture, cannot be preserved [41,42]. In terms of finger-vein images, it is important to restore the high-level features of restored output image similar to those of the original image, however, restoring low-level features is important as well, because vein patterns and texture are slightly different for each class, and performance can be varied due to differences in low-level features during recognition. Because of these reasons, unlike the original DeblurGAN that applied perceptual loss by extracting feature maps from the middle layer of the ImageNet pretrained VGG-19, in this study, feature sets are extracted from the generated image and target image in the first residual block (conv2\_x) using the ImageNet pretrained ResNet-34 [32] model, respectively, and the difference between the two feature sets is applied as a perceptual loss. In a typical neural network, vanishing gradient and explosion occur as the layer gets deeper, eventually resulting in performance degradation. In ResNet, however, this problem is solved by applying a residual learning method. The residual block applying the residual learning method is trained so that identity mapping *F*(*x*) + *x* that is mapping between output *F*(*x*) of the weight layer and output *x* of the layer just before the weight layer, and plain layer output *H*(*x*) are the same (*H*(*x*) = *F*(*x*) + *x*). From the characteristics of the residual block that identity mapping the output information of the previous layer to the next layer, we inferred that low-level features such as color, corner, edge, and texture of the finger-vein can be preserved during restoration training. For this reason, a perceptual loss is applied from the conv2\_x layer of ResNet-34 instead of the original VGG-19.

3.3.4. Summarized Differences between Original DeblurGAN and Proposed Modified DeblurGAN

The differences between the original DeblurGAN and the proposed modified Deblur-GAN are as follows.

finger-vein restoration. Hence, a perceptual loss was applied to the first residual block (conv2\_x) using the ImageNet-pretrained ResNet-34 in the modified DeblurGAN.

### *3.4. Finger-Vein Recognition by Deep CNN*

In this study, the difference image between registered (enrolled) and recognized images was used as an input for CNN-based finger-vein recognition. An image differencing method determines the changes in images where the differences are determined by calculating the pixel differences, and a new image is then created based on the calculation results [44]. Thus, an image differencing method reacts sensitively to the changes in images. For the finger-vein datasets used in this study, if the same class images are used, the pixel difference between the two images is small. So, in general, a pixel value with a low difference image, that is, an image with many black areas is an output. Whereas in the case of other classes, since the pixel difference between the two images is large, the difference image has generally a high pixel value, that is, an image with many bright areas is output. An image differencing method has the advantage of expressing the characteristics of genuine and imposter matching with one output image. Here, genuine matching refers to matching when the input image and the enrolled image are the same class, and imposter matching refers to matching when the input image and the enrolled image are the different class. The finger-vein datasets used in this study have a high similarity of vein patterns between intra-class, but a low similarity between inter-class. Therefore, the finger-vein recognition performance can be verified in the difference image. The examples of finger-vein difference images generated from the dataset used in this study are shown in Figure 7c,f.

**Figure 7.** Difference images of registered and input images. (**a**) Registered image, (**b**) input image of same class as registered image, (**c**) difference image of (**a**,**b**), (**d**) registered image, (**e**) input image of different class as registered image, and (**f**) difference image of (**d**,**e**).

The generated difference image is then used as an input to deep CNN. DenseNet-161 [45] is used to recognition of finger-vein images. DenseNet adopts dense connectivity in which the feature maps of a previous layer are concatenated in the current layer.

$$\mathbf{x}\_{l} = H\_{l}([\mathbf{x}\_{0}, \mathbf{x}\_{1}, \dots, \mathbf{x}\_{l-1}]) \tag{5}$$

Equation (5) represents dense connectivity, where [*x*0, *<sup>x</sup>*1,..., *xl*−1] means the feature map concatenation from layers 0 to *l* − 1. A dense block performs feature map concatenation of the previous and the current layer and transfers the concatenated feature maps to the following layer. *Hl* is a composite function and is composed of batch normalization [46], ReLU [33], and a convolution layer. Generally, as the network becomes deeper, the number of channels of feature maps caused by dense connectivity increases, resulting in an increased number of network parameters. To mitigate the increasing parameters, a bottleneck layer is added to the dense block of DenseNet. As a result, utilizing the bottleneck structure reduces computational costs. However, the output of a dense block concatenates all layers within the block. As the layer gets deeper or the number of layers in the dense block increases, the size and depths of the feature map increase enormously. To solve this problem, a transition layer was added between the dense blocks to reduce the size and depths of the feature maps. The transition layer cuts the number of feature map depths by half through 1 × 1 convolutional computation and reduces width and height by half using 2 × 2 average pooling. In addition, by specifying a growth rate, DenseNet controls the number of output feature map channels. Dense block outputs the feature map at the size of the designated growth rate. In this research, the growth rate is set to 48.

In this study, for finger-vein recognition, the DenseNet-161 pretrained with the ImageNet database [47] is fine-tuned with the finger-vein training data. Difference images are used for the training and testing process, and these images are created using the output restored images by the proposed modified DeblurGAN. The number of output classes of DenseNet-161 is set to 2, genuine matching and imposter matching. The criterion for this is based on the output score obtained from the last layer of the DenseNet. With respect to the threshold of the equal error rate (EER) of genuine and imposter matching distributions of the CNN output score obtained from the training data, it is determined as genuine matching if the CNN output score of the testing data is below the threshold. And imposter matching is determined if the output score is greater than the threshold. The EER is the rate of error at the point where the false rejection rate (FRR) which is the error rate of falsely rejecting genuine matching as an imposter matching and the false acceptance rate (FAR) which is the error rate of falsely accepting imposter matching as genuine matching are equal.

### **4. Experimental Results**

### *4.1. Two Open Databases for Experiments*

In this study, experiments were conducted using two types of open finger-vein databases, SDUMLA-HMT finger-vein database [31] and session 1 images from the Hong Kong Polytechnic University finger-image database version 1 [29]. In SDUMLA-HMT finger-vein database, 6 images from the ring, middle, and index finger from both hands were obtained respectively, from 106 individuals, a total of 3816 images were obtained (2 hands × 3 fingers × 6 images from 106 individuals). In session 1 from the Hong Kong Polytechnic University finger-image database version 1, 6 images from the middle and index finger images were obtained respectively, from 156 individuals, a total of 1872 images were obtained (2 fingers × 6 images from 156 individuals). In this study, the finger-vein database of the SDUMLA-HMT is referred to as SDU-DB, and the session 1 finger-image database version 1 of the Hong Kong Polytechnic University is referred to as PolyU-DB. Figure 8 shows examples from the same finger for PolyU-DB and SDU-DB. The image resolution of SDUMLA-HMT is 320 × 240 pixels, and that of the Hong Kong Polytechnic University finger-image database is 513 × 256 pixels.

SDU-DB consists of 636 classes, whereas PolyU-DB consists of 312 classes. All experiments adopted two-fold cross-validation. Through the two-fold cross-validation method, data of the same class were not used for training and testing (open-world setting). The average accuracy measured through two-fold cross-validation was adopted as the final recognition accuracy.

**Figure 8.** Images obtained from the same finger. (**a**) SDU-DB and (**b**) PolyU-DB.

### *4.2. Motion Blur Datasets for Finger-Vein Image Restoration*

In the case of PolyU-DB and SDU-DB, which are open databases used in this study, motion blurred finger-vein datasets were not constructed. Therefore, to proceed with this study, a motion blurred finger-vein database was constructed by applying motion blurring kernels to the two open databases. When constructing the database, non-uniform (random) motion blurring kernels were applied instead of uniform motion blurring kernels to closely resemble the actual environment. For the random motion blurring kernels, the method proposed by Kupyn et al. [12] was used. Figures 9 and 10 show original and generated motion blurred images of SDU-DB and PolyU-DB.

### *4.3. Data Augmentation and Experimental Setup*

The datasets used in this study do not contain enough images to train a deep CNN, which would result in overfitting. To solve this problem, a data augmentation method was applied to increase the number of training data. For this method, 5 pixel shifting was applied for each image based on 8 directions in a combination of the top, bottom, left, and right. Therefore, each image was increased to 9 times including the original image. Table 4 presents the descriptions of original and augmented data from PolyU-DB and SDU-DB datasets. From the data augmentation, 54 images were generated that increased 9 times from 6 images per class. When training DenseNet-161 for finger-vein recognition, only 1 image among 54 augmented images was selected as an enrolled image, and the other images were used as input images. A difference image was generated using the enrolled image and input image to determine genuine and imposter matching. In the case of SDU-DB, the number of imposter matching was 317 times that of genuine matching, and it was 155 times that of genuine matching for the PolyU-DB. When training this data as it is, a bias on the majority class occurs due to data imbalance. In order to solve this problem, when genuine matching data is augmented with the same number as imposter matching data, training time is increased due to a large number of data, and an overfitting problem for genuine matching data can occur. Therefore, in this study, we applied a random selection method for the imposter matching data. Augmentation and random selection methods were applied to both SDU-DB and PolyU-DB in the same manner, but only to the training data. The original images that were not augmented were used as the testing data.

**Table 4.** Descriptions of experimental databases by data augmentation.

The training and testing were performed on a desktop computer equipped with NVIDIA GeForce GTX 1070 graphics processing unit (GPU) [48] and Intel® Core™ i7-9700F CPU with 16 GB RAM.

### *4.4. Training of Modified DeblurGAN Model for Motion Blur Restoration*

For the training parameter of modified DeblurGAN, the max epoch was set to 100, the mini-batch size was set to 4, and the learning rate was set to 0.0005. Adaptive moment estimation (Adam) optimization [49] was used for the generator and discriminator to train the modified DeblurGAN. Figures 11a,b and 12a,b show the graphs of training loss of the proposed modified DeblurGAN according to the epoch for SDU-DB and PolyU-DB, respectively. The loss values converged as the training progresses, confirming that the proposed modified DeblurGAN was trained sufficiently, as shown in the figures. The trained model with excessive larger number of epochs usually causes the model overfitting. Therefore, we used 10% of training data as validation set which was not used as training. With the trained model of each epoch, the accuracies of validation set was measured, and

the model which showed the best validation accuracy was selected for measuring testing accuracy with testing data. We included the validation performances with validation set in Figures 11c and 12c, which confirms that our model was not overfitted with training data.

**Figure 11.** Training and validation loss graphs of the modified DeblurGAN (SDU-DB): training loss graphs of (**a**) generator and (**b**) discriminator. (**c**) Validation loss graphs of generator and discriminator.

### *4.5. Training of DenseNet-161 for Finger-Vein Recognition*

A stochastic gradient descent (SGD) optimization method [50] was used to train the CNN model for finger-vein recognition. This method involves multiplying a gamma value by the learning rate for every step size at a mini-batch unit to reduce the learning rate, thereby rapidly converging training accuracy and loss. As explained in Section 3.4, DenseNet-161 was used in this study for training and testing. The number of output classes was set to two (authentic and imposter-matching), the number of max epochs was set to 30. The mini-batch size was set to 4, the learning rate was set to 0.001, the step size was set to 16 epochs, the momentum was set to 0.9, and the gamma value was set to 0.1. All the hyperparameters were determined with training data. In detail, the optimal hyperparameters (with which the highest accuracies of finger-vein recognition were obtained with training data) were selected. The search spaces for the number of max epochs, mini-batch size, and learning rate were 10~50, 1~10, and 0.0001~0.01, respectively.

The search spaces for the step size, momentum, and gamma value are 5~25 epochs, 0.1~1, and 0.1~1, respectively.

**Figure 12.** Training and validation loss graphs of the modified DeblurGAN (PolyU-DB): training loss graphs of (**a**) generator and (**b**) discriminator. (**c**) Validation loss graphs of generator and discriminator.

Figures 13 and 14 show the training loss and accuracy graphs of DenseNet-161, which used a difference image restored by the modified DeblurGAN as input. As shown in the training graphs, training loss converged to nearly zero, whereas accuracy converged to nearly 100, indicating that the CNN model for finger-vein recognition was sufficiently trained.

### *4.6. Testing Results of Proposed Method* 4.6.1. Ablation Studies

As ablation studies, experiments were conducted according to with or without motion blur is applied, and the methods can be largely divided into 4 schemes. Scheme 1 means that DenseNet-161 trained with the original training data without blurring was used to perform finger-vein recognition with the original testing data to measure the EER. Scheme 2 means that DenseNet-161 trained with the original training data was used to perform finger-vein recognition with the motion blurred testing data to measure the EER. Scheme 3 represents that DenseNet-161 trained with the motion blurred training data was used to perform

finger-vein recognition with the motion blurred testing data to measure the EER. Lastly, scheme 4 represents that DenseNet-161 trained with the training data restored with the modified DeblurGAN proposed in this study was used to perform finger-vein recognition with testing data restored using the modified DeblurGAN to measure the EER. As shown in schemes 2 and 3 in Tables 5 and 6, the vein-pattern region and other regions were difficult to distinguish, due to motion blur, resulting in degradation of recognition performance. Also, in all cases, compared with schemes 2 and 3, when training was performed with the training data restored with the modified DeblurGAN, and recognition was performed for the testing data restored with the modified DeblurGAN, the recognition accuracy was the highest in scheme 4.

**Figure 13.** Training accuracy and loss graphs of DenseNet-161 using images restored by proposed modified DeblurGAN (SDU-DB).

**Figure 14.** Training accuracy and loss graphs of DenseNet-161 using images restored by proposed modified DeblurGAN (PolyU-DB).

**Table 5.** Comparison of finger-vein recognition error (EER) with respect to the applicable of a motion blur with SDU-DB (unit: %).

**Table 6.** Comparison of finger-vein recognition error (EER) with respect to the applicable of a motion blur with PolyU-DB (unit: %).

Figures 15 and 16 show the receiver operating characteristics (ROC) curves for the recognition performance of schemes 1–4 of SDU-DB and PolyU-DB, respectively. Here, GAR is calculated as 100—FRR (%). As shown in Figures 15 and 16, in all cases, the recognition performance after restoration with the modified DeblurGAN proposed in this study (scheme 4) was higher than schemes 2 and 3.

**Figure 15.** SDU-DB finger-vein recognition ROC curve for scheme 1–4.

**Figure 16.** PolyU-DB finger-vein recognition ROC curve for schemes 1–4.

In Tables 7 and 8, the recognition performances of the modified DeblurGAN model were compared according to the changes in the perceptual loss based on the features extracted from the various CNN models and layers. For a fair performance evaluation, the same recognition model was used for all cases based on scheme 4 to measure the recognition accuracy. For VGG-19 (original DeblurGAN), features extracted from the third convolution layer before the third max-pooling were used. Moreover, features extracted from the first convolution layer before the third max-pooling were used for VGG-19 (conv3.1). This is a result of reflecting the features extracted from a layer prior to VGG-19 (original DeblurGAN) in the perceptual loss, indicating that VGG-19 (original DeblurGAN) showed better recognition performance. For ResNeXt-101 (conv2), better recognition performance was exhibited over VGG-19 (original DeblurGAN) and VGG-19 (conv3.1) for both experiments. Lastly, for ResNet-34 (conv2\_x), the features extracted from the first residual block (conv2\_x) were applied to a perceptual loss (proposed method), thus exhibiting the best performance in all cases with SDU-DB whereas VGG-19 (conv3.1) shows the better accuracies than other cases with PolyU-DB.

**Table 7.** Comparison of finger-vein recognition error (EER) of restored images in SDU-DB according to the perceptual loss based on the various CNN models and layers (unit: %).

**Table 8.** Comparison of finger-vein recognition error (EER) of restored images in PolyU-DB according to the perceptual loss based on the various CNN models and layers (unit: %).

4.6.2. Comparisons with the State-of-the-Art Methods

For the next experiment, the similarities between the images restored with the stateof-the-art methods and the proposed modified DeblurGAN and the original images were quantitatively evaluated. For a numerical comparison, a signal-to-noise ratio (SNR) [52], peak SNR (PSNR) [53], and SSIM [54] were measured. SNR and PSNR are evaluation metrics based on the MSE between two images. Equations (6)–(8) are mathematical equations of MSE, SNR, and PSNR, respectively.

$$\text{MSE} = \frac{1}{hw} \sum\_{i=0}^{h-1} \sum\_{j=0}^{w-1} \left[ I\_o(i,j) - I\_r(i,j) \right]^2 \tag{6}$$

$$\text{SNR} = 10 \log\_{10} \left( \frac{\frac{\sum\_{i=0}^{h-1} \sum\_{j=0}^{w-1} \left[ l\_s(i,j) \right]^2}{hw}}{MSE} \right) \tag{7}$$

$$\text{PSNR} = 10 \log\_{10} \left( \frac{255^2}{MSE} \right) \tag{8}$$

where *Ir* is the restored image obtained from the state-of-the-art or proposed methods, and *Io* is the original image. *h* and *w* are the height and width of an image, respectively. Equation (9) is the mathematical equation of SSIM:

$$\text{SSIM} = \frac{(2\mu\_o\mu\_r + \mathcal{C}\_1)(2\sigma\_{ar} + \mathcal{C}\_2)}{(\mu\_o^2 + \mu\_r^2 + \mathcal{C}\_1)(\sigma\_o^2 + \sigma\_r^2 + \mathcal{C}\_2)}\tag{9}$$

where *μ<sup>r</sup>* and *σ<sup>r</sup>* are the mean and standard deviation of the pixel values of the restored image, respectively. *μ<sup>o</sup>* and *σ<sup>o</sup>* are the mean and standard deviation of the pixel values of the original image, respectively. *σor* is the covariance of two images, and *C*<sup>1</sup> and *C*<sup>2</sup> are constants to prevent the denominator of each equation from becoming zero. Using the evaluation metrics of Equations (6)–(9), the enhancement quality of our proposed method and that of the state-of-the-art was numerically evaluated as shown in Tables 9 and 10. As shown in Tables 9 and 10, SRN-DeblurNet shows the higher values for PSNR, SNR, and SSIM compared to our modified DeblurGAN. That is, the qualities of restored images by SRN-DeblurNet are more similar to those of original ones than those by our method. However, the recognition accuracies by our method are higher than those by SRN-DeblurNet as shown in Tables 11 and 12. That is because the additional noises are included in the restored image and the features similar to the original features cannot be restored by SRN-DeblurNet, which causes the degradation of recognition accuracies although the qualities of restored images are similar to those of original ones.

**Table 9.** Comparisons of blur restoration by using the state-of-the-art methods and proposed modified DeblurGAN with PolyU-DB.

**Table 10.** Comparisons of blur restoration by using the state-of-the-art methods and proposed modified DeblurGAN with SDU-DB.

**Table 11.** Comparisons of finger-vein recognition error (EER) by using the state-of-the-art restoration models and proposed methods with SDU-DB (unit: %).

**Table 12.** Comparisons of finger-vein recognition error (EER) by using the state-of-the-art restoration models and proposed methods with PolyU-DB (unit: %).

Figure 17 shows examples of the finger-vein images restored by state-of-the-art methods and the modified DeblurGAN. For the next experiment, finger-vein recognition performances were compared using the images restored by the modified DeblurGAN and those restored by the state-of-the-art restoration methods for SDU-DB and PolyU-DB, as shown in Tables 11 and 12. For the comparative experiment, the same recognition model was used for a fair performance evaluation to measure the recognition accuracy using the scheme 4 method of Tables 5 and 6. As shown in Tables 11 and 12, finger-vein recognition performance was higher than the existing state-of-the-art restoration methods, when the restoration was performed using the modified DeblurGAN method.

**Figure 17.** Examples of restored images using the state-of-the-art methods and the proposed modified DeblurGAN: (**a**) original images, (**b**) motion blurred images, and the restored images by (**c**) original DeblurGAN, (**d**) DeblurGANv2, (**e**) SRN-DeblurNet, and (**f**) proposed modified DeblurGAN.

Figure 18a,c are the result of authentic and imposter matching prior to restoration, which provide incorrect matching results caused by modified vein patterns and texture information due to motion blur. Authentic matching was falsely rejected as imposter matching, whereas imposter matching was falsely accepted as authentic, thus decreased the recognition performance. Figure 18b,d are the results of correct matching by restoring the incorrect matching problem in (a) and (c) by the modified DeblurGAN. Authentic matching was classified as correct acceptance, and imposer matching was classified as correct rejection.

**Figure 18.** Correct recognition examples after restoring motion blur. (**a**) Incorrect genuine matching before restoring motion blur, (**b**) correct genuine matching after restoring motion blur, (**c**) incorrect imposter matching before restoring motion blur, and (**d**) correct imposter matching after restoring motion blur. From the left, examples in (**a**–**d**) present the registered, input, and difference images, respectively.

Figure 19 is an example of incorrect authentic matching and incorrect imposter matching despite the restoration method proposed in this study is applied. In the case of incorrect authentic matching, the difference in motion blur between the same classes is so severe that it is recognized as an imposter even after restoration, resulting in incorrect matching. In the case of incorrect imposter matching, the enrolled image and the input image appear

similarly in dark shades, and the vein pattern is not clearly visible, so it recognized as authentic even after restoration, resulting in incorrect matching.

**Figure 19.** Incorrect recognition examples after restoring motion blur. (**a**) Incorrect genuine matching before restoring motion blur, (**b**) incorrect genuine matching after restoring motion blur, (**c**) incorrect imposter matching before restoring motion blur, and (**d**) incorrect imposter matching after restoring motion blur. From the left, examples in (**a**–**d**) present the registered, input, and difference images, respectively.

### *4.7. Processing Time of Proposed Method*

For the next experiment, the inference time of the modified DeblurGAN proposed in this study and DenseNet-161 for the finger-vein recognition method was measured. The measurements were taken on the desktop described explained in Section 4.3 and the Jetson TX2 embedded system [55] shown in Figure 20. The reason for measuring using the embedded system is that on-board edge computing, which operates as an embedded system attached to the entrance door, is involved for most access-controlled type finger-vein recognition systems. Thus, it must be verified that on-board computing is feasible on the system proposed. Jetson TX2 has an NVIDIA PascalTM-family GPU

(256 CUDA cores), with 8-GB memory shared between the CPU and GPU, and 59.7-GB/s of memory bandwidth. It uses less than 7.5 watts of power. As presented in Table 13, in the case of the method proposed in this study, the recognition speed for one image was 16.2 ms on a desktop computer and 232.3 ms on the Jetson TX2 embedded system. This corresponds to 61.72 frames/s (1000/16.2) and 4.3 frames/s (1000/232.3), respectively. The processing time on the Jetson TX2 embedded system was longer than the desktop computer, due to limited computing resources. However, through the experiment, it was confirmed that the proposed method is applicable to an embedded system having limited computing resources.

**Figure 20.** Jetson TX2 embedded system.

**Table 13.** Comparisons of processing speed by proposed method on desktop computer and embedded system (unit: ms).

*4.8. Analysis of Feature Map*

4.8.1. Class Activation Map of Restored Image

Figure 21 shows the result of visualizing each class activation map [56] based on the original images and those restored by the proposed modified DeblurGAN in each layer of DenseNet-161. The location from which the class activation map is output is the 1st convolutional layer, the 1st transition layer, the 2nd transition layer, the 3rd transition layer, and the last dense block layer from top to bottom. Figure 21a,b show examples of authentic (genuine) and imposter matching. The left and middle images in (a) and (b) are the original and restored images, respectively. Important features are represented in red, whereas insignificant features are represented in blue in the class activation map. Therefore, if the red and blue regions of the two images appear to be similar, it generally indicates that the two images have similar characteristics. As shown in Figure 21a, in authentic matching, class activation occurs in a similar location of the original image and restored image. Accordingly, it was confirmed that the motion blurred finger-vein image was effectively restored and correct acceptance is possible. As shown in Figure 21b, with

imposter matching, class activation occurs in different locations in the original and restored image, implying that correct rejection is possible.

**Figure 21.** *Cont.*

**Figure 21.** Comparisons of the class activation maps between the original and restored images. (**a**,**b**) are examples of authentic and imposter images, respectively. Images on the left of (**a**,**b**) are the original images, whereas those on the middle are the restored image by proposed modified DeblurGAN. In addition, the images on the right of (**a**,**b**) are the subtracted ones of the middle image from the left one. For both (**a**,**b**), the images from top to bottom are the class activation maps output from the 1st convolutional layer, the 1st transition layer, the 2nd transition layer, the 3rd transition layer, and the last dense block.

In addition, we included the subtracted CAM outputs of restored image from original (motion blurred) one in the right images of Figure 21a,b. The reasons of such differences in the subtracted CAM outputs are that the positions of important finger-vein features extracted are different in original and restored images. Nevertheless, the case of authentic matching (same class) shows the smaller differences as shown in the right image of the last row of Figure 21a compared to that of imposter matching (different classes) in the right image of the last row of Figure 21b. In addition, the reasons of such differences in the subtracted CAM outputs are that the important features of finger-vein can be newly extracted from the restored image (red color in the middle images of Figure 21a,b). However, they cannot be extracted from vein areas in original (motion blurred) image (red color in the left images of Figure 21a,b) due to the indistinctive vein patterns caused by motion blurring, but they are extracted from the other skin areas except for vein regions.

### 4.8.2. Feature Maps of Difference Image

Second, similar to Figure 21, the feature maps of DenseNet-161 were analyzed according to the layer depth in which the difference image between the restored enrolled and restored recognized image as input. The input of DenseNet-161 is the finger-vein image restored by the modified DeblurGAN. As the feature map dimension is too large, the feature maps presented in Figure 22 are each channel's output. Figure 22 presents the examples of the feature maps extracted from genuine and imposter matching images in several layers of DenseNet-161. Examples in Figure 22a–e are the feature maps extracted from the 1st convolutional layer, the 1st transition layer, the 2nd transition layer, the 3rd transition layer, and the last dense block, respectively. In addition, Figure 22f is the 3-dimensional feature map images created by averaging the feature map values of Figure 22e. The top and bottom images in Figure 22 show authentic and imposter matching, respectively.

(**a**)

(**b**)

**Figure 22.** *Cont.*

(**c**)

(**d**)

**Figure 22.** *Cont.*

**Figure 22.** Feature maps extracted from a genuine matching image and an imposter matching image from several layers of the DenseNet-161. (**a**) Feature maps from the 1st convolutional layer; (**b**) feature maps from the 1st transition layer; (**c**) feature maps from the 2nd transition layer; (**d**) feature maps from the 3rd transition layer; (**e**) feature maps from the last dense block, and (**f**) 3D feature maps created by averaging feature map values of (**e**). Upper and lower examples in (**a**–**f**) represent genuine matching feature maps and imposter matching feature maps, respectively.

As shown in Figure 22, abstract features were extracted as the layer became deeper. For example, low-level features, such as lines and corners of the original image, were maintained in Figure 22a, whereas, in Figure 22e, only the abstracted features remained, and shape information mostly disappeared. As shown in Figure 22a–e, the feature maps of authentic and imposter matching do not seem to have a significant difference. However, as shown in Figure 22f, although the changes in the 3-dimensional feature map values drawn by calculating the average of feature map values for the authentic matching results from a step before the classification layer were mostly flat, the results of imposter matching showed that the changes in the feature-map values were greater than those of authentic matching. Therefore, the difference in the CNN feature maps of authentic and imposter matching by the proposed method was confirmed.

### **5. Conclusions**

In this study, a motion blurred finger-vein image was restored to solve the problem of deterioration of finger-vein recognition performance due to motion blur, and a recognition method using deep CNN was studied to evaluate the performance of the restored image. A modified DeblurGAN was proposed by modifying the original DeblurGAN, which was a restoration model. Using two open databases, the recognition error rate was lower when recognition was performed using the restoration method proposed in this study than when images were not restored. Furthermore, based on the comparative experiments using various state-of-the-art restoration models, the proposed method was more effective in restoring an image from motion blur and had more improved recognition performance. Also, based on the analysis of class activation maps and feature maps, it was confirmed that the proposed modified DeblurGAN sufficiently maintained the effective characteristics for classifying authentic and imposter matching. However, as mentioned in Figure 19, it was confirmed that incorrect matching cases occurred despite the proposed restoration method. Therefore, in future studies, a method of increasing restoration and recognition performance by overcoming the extreme difference in motion blur in intra-class and reducing the degree of similarity between inter-classes will be studied. In our research, we used the previous methods [27,29,30] for the ROI detection of finger region, and just focused on the restoration of motion-blur by our proposed modified DeblurGAN and finger-vein recognition by our CNN with the selected ROI. That is because the performance analysis is difficult if both the ROI detection and feature extraction of finger-vein are affected by motion blurring. Therefore, we assume that the ROI without motion blurring is correctly detected by the previous methods [27,29,30], and we only consider that the detected ROI is motion blurred. We would research the motion blurring effect on the boundary detection of ROI in future work.

If the enrolled and recognized images are captured from different camera settings, the performance of finger-vein recognition based on image difference can be affected. However, the enrolled and recognized images are captured from the same capturing device including same camera setting in usual cases of actual finger-vein recognition system. In addition, in this case, the recognition based on image difference showed the better accuracies than those based on original image with the extracted feature vector [20]. Therefore, we use this scheme of image difference for recognition because we mainly focused on the restoration of motion blurring by proposed modified DeblurGAN. We would research the recognition method with the enrolled and recognized images captured from different camera settings in future work.

People usually put their finger on the device with some guiding bar in the actual finger-vein acquisition device (with fixed finger direction) [29,31]. Therefore, there exist only the limited variations of the horizontal and vertical translation and in-plane rotation in the captured finger-vein image. Our data augmentation method aims at covering these individual variations, and it can reduce the recognition error (false rejection case). However, horizontal and vertical mirroring does not happen in the case of a finger-vein image acquisition of the actual capturing device. Therefore, the mirroring generates the images of different classes, which increases the complexity of training data and difficulties of model training. As shown in [57,58], singular value decomposition (SVD) can generate the images of various styles, which can also produce the images of different classes, and it can also increase the complexity of training data and difficulties of model training. Therefore, we use our simple data augmentation method. In future work, we would research the various data augmentation method including SVD and mirroring.

Also, the application of the proposed motion blur restoration method to other biometric modalities, such as iris, face, and palm-vein recognition, will be examined. Moreover, a lighter model that can shorten the processing time will be studied. In future work, we would also research the method with the cases of two open databases combined. In addition, as a future work, we would introduce different types of blurring to the images and develop a generic solution.

**Author Contributions:** Methodology, J.C.; Conceptualization, J.S.H.; Validations, M.O., S.G.K.; Supervision, K.R.P.; Writing—original draft, J.C.; Writing—review and editing, K.R.P. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was supported in part by the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT (MSIT) through the Basic Science Research Program (NRF-2021R1F1A1045587), in part by the NRF funded by the MSIT through the Basic Science Research Program (NRF-2020R1A2C1006179), and in part by the MSIT, Korea, under the ITRC (Information Technology Research Center) support program (IITP-2021-2020-0-01789) supervised by the IITP (Institute for Information & Communications Technology Planning & Evaluation).

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Acknowledgments:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

### **References**

MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel. +41 61 683 77 34 Fax +41 61 302 89 18 www.mdpi.com

*Sensors* Editorial Office E-mail<EMAIL_ADDRESS>www.mdpi.com/journal/sensors

Academic Open Access Publishing

www.mdpi.com ISBN 978-3-0365-7603-9


**3. Results**

### **3.1 Occurrence of NCSE**

Six patients suffered from CSE; only one of them who showed later NCSE EEG features and was included in the study. Twenty patients presented NCSs; 30% of them (n = 6) responded to ASDs and did not develop NCSE on cEEG monitoring;

**49**

**Table 3.**

**Table 2.**

hemorrhage)

*Nonconvulsive Status Epilepticus in Patients with Altered Mental Status Admitted to Hamad…*

they were also excluded from the study; the rest (70%, n=14) developed later NCSE

NCSE group: 250 patients with AMS or coma underwent cEEG monitoring. Sixty-two patients were excluded (see reasons above and patient selection). In total, 65 patient responded to the criteria of NCSE (**Table 2**). The occurrence rate of

The control group consisted of 185 patients with AMS or coma in which cEEG monitoring did not show any features of NCSE. **Table 2** shows the demographic and clinical features of NCSE and control subjects. Only age and presence of subtle motor phenomena differed between the two groups; the NCSE patients were relatively younger and displayed subtle motor phenomena more often. As for etiology and comorbid states, a history of previous seizures and presence of cortical dysplasia were significantly more common in the NCSE group (**Table 3**). Other etiologies were not informative. Head injury, stroke, and status postcardiac arrest were frequently encountered in accident and emergency patients with NCSE; CT head done in 52

**Variable NCSE (n 65) Controls (n 185) P value** Age 45.7 ± 19 52.3 ± 15.8 0.001 Gender M = 37/F = 28 M = 101/F = 84 0.75 Unresponsive/somnolent 11 (17%) 46 (25%) 0.19 Acute confusion 7 (11%) 18 (10%) 0.81 Severely decreased level of consciousness 20 (31%) 61 (33%) 0.74 Stupor/coma 27 (42%) 60 (32%) 0.23 Subtle motor phenomena 12 (18%) 8 (4%) 0.001

**Variable NCSE (n 65) Controls (n 185) P value**

Status post cardiac arrest 15 (23%) 35 (19%) 0.59 Head injury 8 (12%) 34 (18%) 0.34 Previous seizures (uncontrolled) 12 (18.4%) 4 (2%) 0.001 Cortical dysplasia 3 (4.6%) 0 0.02 Sepsis 3 (4.6%) 7 (3.8%) 1.00 Hepatic encephalopathy 1 (1.5%) 3 (1.6%) 1.00 End stage renal disease, post renal transplant 2 (3%) 11 (6%) 0.37 Intoxications 0 8 (4.3%) 0.12 Hypertensive encephalopathy 1 (1.5%) 6 (3.2%) 0.68 Personality disorder 1 (1.5%) 3 (1.6%) 1.0 Unknown 3 (4.6%) 7 (3.8%) 1.0

16 (25%) 67 (36%) 0.09

*Note: P values are calculated using Chi-square tests and student t tests wherever appropriate.*

*Note: P values are calculated using Chi-square tests or Fisher's exact test wherever appropriate.*

*Characteristics of patients with NCSE and controls.*

Stroke (hemorrhagic, ischemic, subarachnoid

*Etiology of patients with NCSE and controls.*

during cEEG monitoring. These patients were included in the study.

**3.2 Characterization of NCSE and the control group**

*DOI: http://dx.doi.org/10.5772/intechopen.83580*

NCSE was 65/250 (26%).

## *Nonconvulsive Status Epilepticus in Patients with Altered Mental Status Admitted to Hamad… DOI: http://dx.doi.org/10.5772/intechopen.83580*

they were also excluded from the study; the rest (70%, n=14) developed later NCSE during cEEG monitoring. These patients were included in the study.

NCSE group: 250 patients with AMS or coma underwent cEEG monitoring. Sixty-two patients were excluded (see reasons above and patient selection). In total, 65 patient responded to the criteria of NCSE (**Table 2**). The occurrence rate of NCSE was 65/250 (26%).