**1. Introduction**

82 Electrical Generation and Distribution Systems and Power Quality Disturbances

[54] N. Chayawatto, K. Kirtikara, V. Monyakul, C Jivacate, D Chenvidhya, "DC–AC

[55] Lennart Ljung, "System identification : theory for the user", Upper Saddle River, NJ,

[56] Nelles, Oliver, Nonlinear system identification : from classical approaches to neural

[57] N.Patcharaprakiti and et al., "Nonlinear System Identification of power inverter for grid

phenomena", Renewable Energy 34 (2009) 2536–2544.

[58] Lennart Ljung, System Identification Toolbox User Guide, 2009

[59] IEEE 1159, 2009 Recommended practice for monitoring electric power quality.

Prentice Hall PTR, 1999.

networks and fuzzy models, 2001

August 2010, Bangkok, Thailand.

switching converter mode lings of a PV grid-connected system under islanding

connected Photovoltaic System based on MIMO black box modeling", GMSTECH 2010 : International Conference for a Sustainable Greater Mekong Subregion, 26-27

> A basic component of the power quality generally and electricity supply in particular is the management of maintenance actions of electric transmission and distribution networks (EDS). Starting from this fact, the chapter develops a mathematical model of external interventions upon a system henceforth, called Renewal Processes. These are performed in order to establish system performance i.e. its availability, in technical and economic imposed constraints. At present, the application of preventive maintenance (PM) strategy for the electric distribution systems, in general and to overhead electric lines (OEL) in special, at fixed or variable time intervals, cannot be accepted without scientifically planning the analysis from a technical and economic the point of view. Thus, it is considered that these strategies PM should benefit from mathematical models which, at their turn, should be based, on the probabilistic interpretation of the actual state of transmission and distribution installations for electricity. The solutions to these mathematical models must lead to the establishment objective necessities, priorities, magnitude of preventive maintenance actions and to reduce the life cycle cost of the electrical installations (Anders et al., 2007).

#### **1.1 General considerations and study assumptions**

Based on the definition of IEC No: 60300-3-11 for RCM: "method to identify and select failure management policies to efficiently and effectively achieve the required safty, availability and economy of operation", it actually represents a conception of translating feedback information from the past time of the operation installations to the future time of their maintenance, grounding this action on:

So, Reliability Based Maintenance (RCM) implies planning the future maintenance actions ( T+ ) based on the technical state of the system, the state being assessed on the basis of the estimated reliability indices of the system at the planning moment ( 0 T ). At their turn, these reliability indices are mathematically estimated based on the *record of events*, that is, based on previously available information, related to the behaviour over period ( T− ), i.e. to the ( 0 T ) moment, concordant with Figure 1.

Reliability Centered Maintenance Optimization of Electric Distribution Systems 85

In theory of reliability, the concept of the *wear* has a wider meaning than in ordinary language. In this context, the wear includes any alteration in time of characteristics of reliability, for the purposes of worsening or improving them (Catuneanu & Mihalache 1983). Considering the function of the reliability of equipment for a mission with a duration *x,*

( ) ( )

R t,t x R t

From a the mathematical point of view, we can express the wear through *failure rate (FR)* of the equipment. From the relationship (2) and from the defining relationship of failure rate

( )

If the failure rate of the equipment with positive wear increases in time, then these systems will be called IFR type systems (Increasing Failure Rate) and if the failure rate of the equipment with negative wear decreases in time, these systems will be called DFR type

An equipment is of *NBU degradable type* if the reliability function associated to a mission of the duration *x,* initialized at the *t* age of equipment, is less than the reliability function in

From the previous definition, results that a degradable equipment which was used is

The notion of degradable equipment is less restrictive than that of equipment with positive wear, which supposed the decreasingd character of the function *Rt t x* ( ) , + with *t* age of

x 0 x 0 R t R t x R t,t x z t lim lim → → xRt x

( ) () ( )

( ) Rt x

+ = (2)

( )

R t,t x R t ; t,x 0 ( ) () +< ≥ (4)

−+ + = = ⋅ (3)

+

reliability and of the notions of: *wear, degradable system and renewal actions*.

**1.3 Modelling of systems wear**

we obtain:

equipment.

initialized at the *t* moment, of the type:

systems (Decreasing Failure Rate).

inferior that to a new equipment.

equipment, its reliability decreases with its age.

equipment, its reliability increases with its age.

interval *(0, t)*, regardless of the age and duration of *x* mission.

Fig. 1. Assessment and planning times of the RCM

Even if in case of the OEL, the two actions are apparently independent, as they take place at different times, they influence each other through the model adopted for each of them. Thus:

Accepting that the expression of the availability at a time *t,* the relationship given by the form (Baron et al., 1988):

$$\mathbf{A(t)} = \mathbf{R(t)} + \left\lfloor \mathbf{1} - \mathbf{R(t)} \right\rfloor \cdot \mathbf{M(t')} \tag{1}$$

where:

A(t) – Availability;

R(t) – Reliability;

M(t') – Maintainability

It can be inferred that the availability of an element or of a system is determined by two probabilities:

Thus, for components of EDS which are submissions by RCM actions, a distinctive study is required to model their availability under the aspect of modelling those two components:

As the mathematical models of the two components R(t) & M(t') interdependent, we will continue by exposing some considerations regarding the establishment of M(t'), and section 3 will be devoted to developing the component R(t) of the EDS.

#### **1.2 EDS-Degradable systems subjected to wear processes**

EDS in general and the OEL in particular, contain parts with mechanical and electrical character and their operation is directly influenced by different factors, being constantly subjected to the requirement of mechanical, electrical, thermal processes, etc. The literature shows the percentage of causes which determin interruptions of OEL, (Anders et al., 2007).

#### **1.3 Modelling of systems wear**

84 Electrical Generation and Distribution Systems and Power Quality Disturbances

Even if in case of the OEL, the two actions are apparently independent, as they take place at different times, they influence each other through the model adopted for each of them. Thus: - the analytical expression of the reliability function adopted in while ( T− ), depends directly by the basis of specific physical phenomena (such as wear, failure, renewal,

It can be inferred that the availability of an element or of a system is determined by two

a. the probability that the product to be in operation without failure over a period of time

b. the probability that the element or the system, which fails over time interval (*t*), to be reinstated in operation in time interval (*t'*) ; M(t') *-* is called the *Maintainability function.*  Thus, for components of EDS which are submissions by RCM actions, a distinctive study is required to model their availability under the aspect of modelling those two components: 1. The *reliability function* R(t) of EDS, respectively of the studied component, in this case a

2. The *Maintainability function* M(t'), under the aspect of their specific constructive and

As the mathematical models of the two components R(t) & M(t') interdependent, we will continue by exposing some considerations regarding the establishment of M(t'), and section

EDS in general and the OEL in particular, contain parts with mechanical and electrical character and their operation is directly influenced by different factors, being constantly subjected to the requirement of mechanical, electrical, thermal processes, etc. The literature shows the percentage of causes which determin interruptions of OEL, (Anders et al., 2007).

A(t) R t 1 R t M t' = + () () ( ) − ⋅ (1)

etc.) of equipment during maintenance actions in time interval ( T+ );

Fig. 1. Assessment and planning times of the RCM

(*t*); R(t) – is called the *reliability function*;

operation conditions, maintenance.

3 will be devoted to developing the component R(t) of the EDS.

**1.2 EDS-Degradable systems subjected to wear processes**

form (Baron et al., 1988):

A(t) – Availability; R(t) – Reliability; M(t') – Maintainability

probabilities:

OEL;

where:

In theory of reliability, the concept of the *wear* has a wider meaning than in ordinary language. In this context, the wear includes any alteration in time of characteristics of reliability, for the purposes of worsening or improving them (Catuneanu & Mihalache 1983). Considering the function of the reliability of equipment for a mission with a duration *x,*  initialized at the *t* moment, of the type:

$$\mathbf{R}\left(\mathbf{t}, \mathbf{t} + \mathbf{x}\right) = \frac{\mathbf{R}\left(\mathbf{t} + \mathbf{x}\right)}{\mathbf{R}\left(\mathbf{t}\right)}\tag{2}$$

From a the mathematical point of view, we can express the wear through *failure rate (FR)* of the equipment. From the relationship (2) and from the defining relationship of failure rate we obtain:

$$\text{var}(\mathbf{t}) = \lim\_{\mathbf{x} \to \mathbf{0}} \frac{\mathbf{R}(\mathbf{t}) - \mathbf{R}(\mathbf{t} + \mathbf{x})}{\mathbf{x} \cdot \mathbf{R}(\mathbf{t})} = \lim\_{\mathbf{x} \to \mathbf{0}} \frac{\mathbf{R}(\mathbf{t}, \mathbf{t} + \mathbf{x})}{\mathbf{x}} \tag{3}$$

If the failure rate of the equipment with positive wear increases in time, then these systems will be called IFR type systems (Increasing Failure Rate) and if the failure rate of the equipment with negative wear decreases in time, these systems will be called DFR type systems (Decreasing Failure Rate).

An equipment is of *NBU degradable type* if the reliability function associated to a mission of the duration *x,* initialized at the *t* age of equipment, is less than the reliability function in interval *(0, t)*, regardless of the age and duration of *x* mission.

$$\mathbb{R}(\mathbf{t}, \mathbf{t} + \mathbf{x}) < \mathbb{R}(\mathbf{t}); \quad \mathbf{t}, \mathbf{x} \ge \mathbf{0} \tag{4}$$

From the previous definition, results that a degradable equipment which was used is inferior that to a new equipment.

The notion of degradable equipment is less restrictive than that of equipment with positive wear, which supposed the decreasingd character of the function *Rt t x* ( ) , + with *t* age of equipment.

A particular problem consist of identifying the type of wear that characterizes an equipment/system, which can be obtained from reliability tests or from the analysis of the moments of failure in operation, which mathematically modelled, give the wear function, denoted by Ts (F) .

$$\mathbf{F(t) = 1 - R(t)}\tag{5}$$

Reliability Centered Maintenance Optimization of Electric Distribution Systems 87

The classification of renewal actions can be performed on several criteria, of which we remind a few: the purpose, timing and costs of their occurrence, distribution and frequency

In this work, we analyze and study only preventive renewal processes, through their

The evolution of an equipment will thus be represented by the succession of renewal moments 1 2 , , ... , *n tt t* , and the intervals between them: 1 2 , , ... , ... *n xx x* presented in the

If it is considered a certain time interval (0, t), the number of PR denoted by *Nt* , performed during this time, is a discrete deterministic process, called preventive renewal process, as the basic component of preventive maintenance, which in turn determine planning, development and the effects of RCM on system. This, requires the development of PR strategies, enabling knowledge of behaviour of renewal equipment, used in the

Studies classify and model the renewal strategies according to technical and economic parameters, which determine these strategies, in two distinct categories: non-periodic and

and last but not least, the effects on the system safety, (Anders et al., 2007).

added to a random process of failure renewals.

modelling influence on the system in the following cases:

Fig. 2. The evolution of an equipment with renewal

periodic, (Catuneanu & Mihalache, 1983), (Andres et al., 2007).

received from operation;

this point of view;

economic ones;

the system as a whole.

development of PM programs.

**2.2 Renewal strategies**

failure renewals.

renewal;

Figure 2.

their strategies. Thus, the random or deterministic strategy of preventive renewals is

According to system reliability T fFt S = ( ) ( ) , the graph function of the wear inscribed in a square with l side, can indicate the type of equipment wear by its form:

Thus, the literature allow for the study maintenance of OEL, the following assumptions:

We mention that all models specific to these assumptions/behaviours of the OEL are dependent on the reliability function of the system in study and their effects model this function.


**4.4. Discussions and recommendation for further research**

602 Numerical Simulation – From Theory to Industry

Fig. 19 illustrates the distortions of the welding structure due to the welding sequences. The initial undeformed configurations were also shown. From Fig. 19, it can be seen the angular distortions occurred in both flanges. It can be further revealed that there was the difference of distortion between the flanges showing that the distortion was unsymmetrical. The maximum value of angular distortion took place on the right flange for all the welding sequences, unless that of WS-2 which took place on the left one. The simulation results obtained also clearly indicate the influence of the welding sequences examined in the present study to the angular distortions of the T-joint fillet weld considering the same boundary conditions appliedin the corresponding FEM models of the T-joint fillet weld.

Furthermore, Table 3 summarizes the vertical displacements and the angular distortions of both flanges due to the welding sequences. The angular distortion differences were also

**Figure 19.** Distortions of the welding structure due to the welding sequences: (a) WS-1, (b) WS-2, (c)

**4.3. Distortions**

shown in Table 3.

WS-3, and (d) WS-4.

From the results, it seems that, for the problem considered in this numerical study, two welding sequences, namely WS-2 and WS-4, have taken the attention. The WS-2, which is called as simple alternating welding, has produced the lowest peak temperature and the smallest longitudinal and transverse residual stresses as well. Meanwhile the WS-4, which is called as multiple crossing welding, has produced the smallest angular distortion and angular distortion difference, although it produced the highest peak temperature.

The information appears to be consistent with respect to the welding sequences performed. The corresponding value of the von Mises stress and the distortion difference produced as shown respectively in Table 2 and 3 indicated this as well. In particular, the results were also in contrast to those of WS-1 and WS-3. Not only did the welding sequences produce high angular distortions, but also they resulted in relatively high values of the von Mises stresses. Furthermore, the distortion results obtained appears to be match with the ones usually found in the welding practice incorporating alternating welding.

Also, considering limited literatures concerning welding simulation of T-joint fillet welds in 3D (Chang & Lee, 2009 and Deng et al., 2007), the results obtained would be very valuable

and useful to welding designers and practitioners, because the results have been describing the predicted or anticipated residual stresses and distortions with respect to the welding sequences, varied from simple to multiple crossing welding. In addition, the assessment of welding performance which can be taken in an efficient and fast manner allows the designers to integrate it in their subsequent design plans.

3D Finite Element Simulation of T-Joint Fillet Weld:

Effect of Various Welding Sequences on the Residual Stresses and Distortions 605

**Author details**

*Surabaya, Indonesia*

Mas Irfan Purbawanto Hidayat

*(ITS), Surabaya, Indonesia*

**Acknowledgement**

acknowledged.

**6. References**

41, 250-258.

New York.

Cambridge.

Eng. , Vol. 195, 6710-6736.

Piping, Vol. 78, 523-538.

Concequences, Pergamon Press Ltd, London.

Stresses, Computers and Structures, Vol. 81, 273-286.

*Department of Ocean Engineering, Institut Teknologi Sepuluh Nopember (ITS),*

High-Performance Alloys, ASM International USA.

*Department of Materials and Metallurgical Engineering, Institut Teknologi Sepuluh Nopember*

Funding provided by Institut Teknologi Sepuluh Nopember (ITS) Surabaya is gratefully

ASM (1990). Metals Handbook Vol. 1, 10th ed. , Properties and Selection: Irons, Steels, and

Bette (1999). Fabrication, Applications Engineering, in: The Welding Engineer's Current

Chang, K. H. & Lee, C. H. (2009). Finite Element Analysis of the Residual Stresses in T-joint Fillet Welds Made of Similar and Dissimilar Steels, Int. J. Adv. Manuf. Technol. , Vol.

Deng, D. , Murakawa, H. & Liang, Wei (2007). Numerical Simulation of Welding Distortion in Large Structures, Comput. Methods Appl. Mech. Eng. , Vol. 196, 4613-4627. Goldak, J. A. & Akhlaghi, M. (2005). Computational Welding Mechanics, Springer, Inc. ,

Grong, O. (1994). Metallurgical Modelling of Welding, The Institute of Materials,

Lindgren, L. E. (2006). Numerical Modelling of Welding, Comput. Methods Appl. Mech.

Masubuchi, K. (1980). Analysis of Welded Structures: Residual Stress, Distortion, and their

Teng, T. L. & Lin, C. C. (1998). Effect of Welding Conditions on Residual Stresses due to Butt Welds, International Journal of Pressure Vessels and Piping, Vol. 75, 857-864. Teng, T. L. , Fung, C. P. , Chang, P. H. & Yang, W. C. (2001). Analysis of Residual Stresses and Distortions in T-joint Fillet Welds, International Journal of Pressure Vessels and

Teng, T. L. , Chang, P. H. & Tseng, W. C. (2003). Effect of Welding Sequences on Residual

Bathe, K. J. (1996). Finite Element Procedures, Prentice Hall, Inc. , New Jersey.

Knowledge ed. 2000, EWE-3/4. 12 pp. 8-11, SLV Duisburg Gmbh, Germany.

Nur Syahroni

Furthermore, the 3D simulation results of T-joint fillet weld may be further used as validation model for 3D welding simulations as well as for other numerical technique implementations such as mesh-less techniques, where no predefined mesh is required to build interpolation of the potential field variables investigated thus reducing cumbersome mesh preparation and increasing the related simulation time.

Moreover, the relationship between the input and output variables of the welding process may be further investigated and optimized using techniques from artificial intelligence (AI) family, such as neural networks and genetic algorithm. For examples, in the single pass GTAW welding method presented in this study, the variables of welding current, voltage, welding speed and welding sequences have been examined, in which more output variables may be also examined, such as the nature and dimensions of weld bead. Thus, much more information and insights can be revealed in such a welding process, which is in turn very useful to optimize the welding process.

It is noted here that the aspects of shrinkage were not discussed in the present paper. The aspects could be also related to the variation of welding speed. Also, it may be interesting if some welding paths in one WS are performed and simulated simultaneously thus allowing the exploitation of symmetry and anti-symmetry boundary conditions in the finite element model. The aforementioned aspects would be the subjects of further investigations.


**6. Conclusions**

Sharing knowledge in publications has a long tradition in scientific research since this is the elemental way of consuming and communicating information and knowledge by human scientists. Within the domain of tribology, the vast amount of available information overcomes the cognitive capacity of humans in terms of efficient aggregation and processing. Therefore, AI provides a lot of potential to support scientists by handling this flood of information. Nevertheless, the way of knowledge sharing within tribology is still mainly based on publications and thus human focused. Descriptions in natural language are vague and insufficient from a formalization perspective. This phenomenon is due to an intended human consumer, for whom the provision of formal sufficient information would result in far too long publications and re-reading the same information over and over again. In contrast, machines need a formal and explicit model for aggregating and processing information. Therefore, if the available amount of information overwhelms the capacity of human processability, the question arises if we better should create representations for sharing information with an AI instead of humans in the future? The answer is: Not necessarily. As pointed out by Gruber [74], "the purpose of AI is to empower humans with machine intelligence". This is referred to as "humanistic AI", an artificial intelligence designed to meet human needs by collaborating with and augmenting people. In terms of an AI empowered tribological knowledge sharing, we introduced a semantic annotation pipeline towards generating knowledge graphs from natural language publications to bridge the gap between a human-understandable and a machine-processable knowledge representation. The pipeline is built upon state-of-the-art NLP methods and is inspired by similar challenges from the biomedical domain. Although we demonstrate the potential of the approach (NER and QA show reliable computational performance scores), further validation of the approach to ensure practical usability is recommended. This includes especially the definition of the specific objective of the extracted information, e.g., for trend studies or identifying research gaps and contradictions within the domain of tribology. Furthermore, the annotation model is currently limited to model tests and is not validated to suit the practical information needs of tribologists. Therefore, user studies for analyzing the capability of information extraction compared with human experts provide possibilities to improve the performance of the approach.

**Author Contributions:** Conceptualization, P.K. and M.M.; methodology, P.K. and R.D.; software, R.D.; validation, R.D. and P.K.; writing—original draft preparation, P.K. and M.M.; writing—review and editing, P.K., M.M., R.D., B.S. and S.W.; visualization, P.K. and M.M.; supervision, B.S. and S.W.; funding acquisition, S.W. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The code for the Semantic Annotation Pipeline is available under https://github.com/snow0815/TriboAnnotation.git, accessed on 14 December 2021.

**Acknowledgments:** P. Kügler, B. Schleich and S. Wartzack gratefully acknowledge the financial support of project WA 2913/22-2 within the Priority Program 1921 by the German Research Foundation (DFG). M. Marian acknowledges the support from Pontificia Universidad Católica de Chile.

**Conflicts of Interest:** The authors declare no conflict of interest.

## **References**


Artificial Intelligence (AI) is the reflection of human cognitive functions from the surroundings acquired by applying algorithms, pattern matching, cognitive computing, and deep learning to achieve specific objectives [88]. The ongoing progress in AI, primarily in the sub-domains of machine learning (ML) and deep learning (DL), have caught the attention of physicians hoping to develop newly integrated, dependable, and potent methods for ensuring standard healthcare in the critical field of cardiology.

Machine learning (ML) is a subset of AI to "teach" computers to analyze huge datasets in a quick, accurate, and efficient manner by using complex computing and statistical algorithms [89]. Supervised ML is more successful in predicting survival compared to the traditional clinical risk scores [90].

The study [91] proved that the accuracy of disease prediction can be increased by using an unsupervised type of ML for obstructive coronary artery disease in nuclear cardiology.

Deep learning (DL) is a supervised ML methodology that relies on neural networks and is known for the automated algorithms required to extract meaningful patterns from data collections [92]. In the medical context, the most widespread deep learning algorithms are artificial neural networks (ANN), multilayer perceptron (MLP), convolution neural networks (CNN/ConvNet), recurrent neural networks (RNN), radial basis function network (RBFN), deep belief networks, and deep neural networks (DNN) [88]. Compared with traditional supervised ML, the real strength of DL is that it is an effective, powerful, and flexible approach to representing complicated raw input data that does not demand manual feature engineering. For instance, while addressing the issue of automated ECG interpretation, early conventional supervised ML techniques depended on human-defined ECG features. In contrast, the modern DL model extracts patterns within raw ECGs to detect sinus rhythm and various other arrhythmias with a performance that equals the result of any cardiologist [93].

The significant areas of cardiac healthcare that can benefit from ML/DL techniques are prognosis, diagnosis, classification, treatment, and clinical workflow. Table 12 presents an overview of different AI algorithms extracted from the literature review on heart disease diagnosis/classification.

**Table 12.** Overview of common ai techniques used in E-Cardiology.

A comparative analysis of different AI techniques frequently used in Smart cardiology for the prognosis/diagnosis of various CVDs is given in Table 13.

As suggested by WHO, by 2030 almost 23.6 million individuals will die from heartrelated causes [94]. CVDs are the main cause, but they can be cured and prevented. To reduce the risk involved, analysis is fundamental. The difficult part is accurate diagnosis [95].

Table 14 summarises the most recent work performed in the field of artificial intelligence related to CVDs.

PCA-KNN—principal component analysis with K-nearest neighbor; NB—naïve Bayes; RF—random forest; SVM—support vector machine; LR—logistic regression; DL—deep learning; ANN—artificial neural networks; BP—backpropagation; Y—yes; S—slow; F—fast; L—low; H—high; N—no.

**Table 14.** Summary of AI-methodologies and data preprocessing techniques identified for E-Cardiology, from different studies.

**Table 14.** *Cont.*

H—high; M—medium; N/A—not available; DL—deep learning; MPNN-BP—multilayer perceptron neural network-backpropagation; RF—random forest; WPE—wavelet packet entropy; CNN—convolutional neural network; CL—K means clustering; ZAS—Z alizadeh; PTB—physikalisch technische bundesanstalt; NB—naive Bayes; FFT—fast fourier transform; DNN—deep neural network; SVM—support vector machine; NN—neural networks; BBNN—block-based neural network; MC—Mayo Clinic; HWT—haar wavelet transform; EMD empirical mode decomposition; RBNN—radial basis function neural network; OCAD—obstructive coronary artery disease; DBN—deep belief networks; ROC—receiver operating characteristic curve; DHCAF—dynamic heartbeat classification; SFS—sequential forward transform; DOST—discrete orthogonal stockwell transform; DOM—difference operation method; CCTA—cardiac CT angiography; PSO—particle swarm optimization; CUDB—Creighton University database; VFDB—ventricular fibrillation database; LS—SVM-least square SVM; OR—online repository; DBN—deep belief networks; ROC—receiver operating characteristic curve; DWT discrete wavelet transform; DHCAF—dynamic heartbeat classification with adjusted features; MCHCNN—multichannel heartbeat convolutional neural network.

#### 5.3.2. Data Preprocessing Techniques in E-Cardiology

This section identifies and evaluates studies that applied data preprocessing techniques in cardiac disease classification. Data Preprocessing (DP) in AI is a critical stage that enhances the quality of data to achieve meaningful insights and is the initial step in the development of an AI model. Conventionally, real-world data is not in an appropriate format and contains errors or outliers. It usually lacks specific attribute values/trends, thus resulting in an inadequate AI model. Data preprocessing solves this problem by cleaning and organizing raw data to tailor it to the needs of building and training AI models. Hence, data preprocessing in AI is a data mining approach that reshapes raw data into a readable format that is readily available for an AI model to meet the high standards of performance [120]. Consequently, the algorithm can easily interpret the data's features.There are four primary ways of data preprocessing, i.e., (1) data cleaning, (2) data integration and formatting, (3) data transformation, and (4) data reduction.

Different preporocessing techniques used in past studies for diagnosing heart disease and other types of arrhythmia are also mentioned in Table 14, along with AI models. This table also lists the task performed by the preprocessing technique.

#### *5.4. RQ 4: What Are the Major Issues and Challenges in Current E-Cardiology?*

After conducting comprehensive research, we identified some significant benefits and major challenges in the field of MIoT to answer our RQ 4. These challenges and benefits have been emphasized on the basis of studies conducted by different researchers in the domain of MIoT and E-Cardiology. Based upon selection and rejection criteria, only valid and reliable papers were selected, as mentioned earlier. We incorporated only the latest benefits and challenges that were found to be unique in the domain of IoT and AI regarding

healthcare and cardiology. A pictorial representation of these benefits and challenges is shown in Figure 4.

**Figure 4.** Benefits and challenges of E-Cardiology.

5.4.1. Benefits of E-Cardiology

*Internet of Things (IoT)* develops a linkage between "things", such as devices, gadgets, vehicles, and sensors. Likewise, the medical Internet of Things (MIoT)-based cardiovascular healthcare system monitors the physical symptoms [5] of cardiac patients at a very reasonable cost. These physical symptoms include temperature, blood pressure (BP), SPO2, and heartbeat, along with ECG [6] and associated numerical measurements. Significant benefits of IoT-based cardiology from various studies are noted in Table 15.

#### **Table 15.** *Cont.*

A comparison of some key factors in Smart cardiac care are shown in Table 16. *Artificial intelligence (AI)* is another significant aspect of E-Cardiology. Until now, AI in cardiac electrophysiology has exhibited promising results. Primary advantages of AI-based cardiology are discussed in Table 17.

**Table 16.** Comparison of the existing evaluation factors in the Smart cardiac care.

#### **Table 16.** *Cont.*

✓—parameter mentioned; ✗—parameter not mentioned.

#### **Table 17.** Key challenges and primary benefits of AI-based cardiology.

#### 5.4.2. Challenges of E-Cardiology

*Internet of Things (IoT)* comes with various challenges as shown in Table 15. In addition to these challenges, ref. [129] proposed some other vital challenges, such as fixation of sensors, body impact on signal propagation, and synchronization, that may affect critical health services such as cardiac care. The MIoT-based healthcare systems are expected to produce a vast amount of data. Moreover, these sensors and devices are linked through networks, thus enabling real-time transmission of data. Therefore, hackers may attempt to target it. Moreover, the timely availability of medical data will affect the patient's life. Consequently, it is crucial to have real-time information with lower latency over the network [124].

*Artificial intelligence (AI)* has brought a revolution in the field of healthcare. It has especially made a great contribution in the domain of cardiac care, such as timely prediction and diagnosis of cardiovascular diseases (CVDs), ECG analysis, and arrhythmia classification. However, despite all these milestones, it carries some challenges. Table 17 describes critical issues and challenges of AI-based cardiology from different past studies.

It is hoped that these challenges can be met and that through MIoT and AI we can achieve new levels of technical and medical standards in the field of cardiac healthcare.

#### **6. Discussion**

Figure 5 shows the distribution of the selected studies (104 papers) according to our four research questions (RQs). The pie chart shows that 47 studies overlap RQ1 and RQ2, whereas 33 studies address RQ3 and 24 address RQ4.

**Figure 5.** Statistical analysis of reviewed papers in terms of formulated RQs.

The outcomes of this systematic literature review suggest a noticeable increase in the research conducted in IoT using artificial intelligence in E-Cardiology. The research activities also incorporate monitoring CCU parameters, ECG analysis, and diagnosis/classification of various heart diseases. This extensive review study reveals that IoT sensors utilized for E-Cardiology are based upon analog sensors, digital sensors, and different wearable sensors or device modules. For fitness tracking and health monitoring activities, a variety of ready-made and wearable watches are also available. E-Cardiology patients can use a variety of wearable gadgets to track various cardiac healthcare characteristics. To establish expert E-Cardiology systems, several communication technologies and protocols for transmissions over a defined range need to be instituted and maintained. Our findings on communication technologies and protocols for the implementation of IoT-based cardiovascular healthcare monitoring and detection systems show that the following technologies provide viable tools to use in MIoT systems: Bluetooth (BL/BLE), ethernet, global system for mobile (GSM), global positioning system (GPS), global packet radio service (GPRS), message queuing telemetry transport (MQTT), short message service (SMS), email, zigbee, transmission control protocol/Internet protocol (TCP/IP), wireless fidelity (Wifi), security protocols, broadband/Internet, cloud, Smart cars, Smart phones/computers, etc. Our findings show that deep learning is often used in cardiac imaging procedures, particularly in echocardiography [88]. Furthermore, CNNs have been evaluated and found useful in calculating coronary artery calcium in cardiac CT angiography [101]. Though deep learning techniques are garnering attention in the field of Smart cardiology, we may infer that instead of depending on a particular AI model, hybrid techniques are expected to

produce better results. From the results of our study, it may also be inferred that SVM was more frequently used in cardiac care than was deep learning. However, in recent years, deep learning has emerged as a more powerful and reliable tool for the detection and diagnosis of various heart diseases. It was also noted that data reduction appears to be a major concern of researchers when applying data mining/data preprocessing approaches to predict CVD. This literature review includes 24 studies devoted to major issues and challenges in E-Cardiology. These studies conclude that a major benefit of MIoT in cardiac healthcare is that it generates timely and accurate data, which results in better healthcare outcomes. One vital advantage of using ML techniques is the ability to fuse various types of data [143]. We also assessed the results and noted that mIoT devices do not possess the requisite data protocols and standards [131]. Therefore, many issues must be addressed to ensure IoT privacy and security [144,145], which is one of the major challenges of the IoT era. Moreover, the continuous monitoring of critical indicators requires reduced energy consumption and a longer battery life [146] to prevent a break of communication. This is also one of the significant challenges of MIoT. The medical data gathered using MIoT is seldom standardized and often fragmented. The data in legacy IT systems are usually generated with incompatible formats. Thus, the great challenge of interoperability needs to be addressed as well [124,147]. However, to move forward in the field of MIoT, fearing AI is no option.

Instead, we should work toward the smooth digitization of healthcare infrastructure [148]. Obviously, various benefits of AI cannot be implemented and utilized correctly without integrating AI into clinical decision making effectively and responsibly [149].

Figure 6 refers to the research work conducted based on the number of papers per annum. It shows the year-wise trends in publications in the field of E-Cardiology. It also indicates that the maximum number of papers selected for this survey was from 2019.

#### *6.1. Gaps, Future Recommendations*

In a future work, all the vital cardiac parameters can be combined with an intelligent cardiac care unit (CCU) to develop a complete picture of E-Cardiology. These parameters/indicators may be comprised of temperature, blood pressure, oxygen saturation, heart rate, and ECG analysis. In addition, in differentiating between normal and abnormal heartbeats, a Smart CCU can also be used to detect QRS complexes in electrocardiographic (ECG) signals to determine the presence of a cardiac malady and different arrhythmias. Integrated and wearable IoT solutions, which address all the necessary cardiac parameters of a heart patient, need to be implemented. The results and accuracy of the devices/sensors used in the development of an IOT-based cardiac system cannot be compromised. The implemented system must be tested, evaluated, and approved under the supervision of cardiologists. Advanced communication technologies, including secure network protocols, must be implemented. Data accessibility features, such as widespread data access, must be possible in a secure environment so that the data confidentiality and integrity are maintained. Ubiquitous access is also an important factor and can be achieved by storing the digital data on a cloud server.

#### *6.2. Limitations of the Review Study*

This review of literature has some limitations. First, many papers were conference proceedings; therefore some parameters remained inaccessible since their authors did not mention them in detail. Second, some of the studies on AI-based IoT architecture in cardiac healthcare could not be located even after following a comprehensive search protocol, such as gray literature and reports that were not published in the databases which we selected for review. Therefore, we suggest that an additional systematic study be conducted to cover the related literature from other important databases.

**Figure 6.** Year-wise trends in publications in the field of E-Cardiology.

#### **7. Conclusions**

This review study outlines a total of 104 primary studies from 2016 to 2021 based on our filtering process for supporting the proposed research. Quality assessment of the selected studies was conducted for the formulated research questions after a rigorous analysis.

This work mentions different sensors and communication technologies being used in cardiology. Moreover, this review study also describes various preprocessing techniques and AI algorithms used in the existing studies to diagnose and classify CVD and ECG analysis. This systematic review also provides comparative analysis of various existing techniques in the field of AI, medical sensors, and communication technologies. Finally, this study targets various advantages and issues indicated in the existing literature in the field of E-Cardiology. The interaction of MIoT and artificial intelligence makes cardiac healthcare more manageable by making various applications, services, communication protocols, third party APIs, and IoT sensors available. E-Cardiology guarantees more privacy and security to the IoT devices which are prone to hackers. Furthermore, AI-based diagnosis of various cardiovascular diseases in E-Cardiology helps save time, enabling cardiologists to focus more on treatment. This systematic work presents a review protocol to analyse how IoT applications assist cardiac healthcare and how various artificial intelligence (AI) models contribute to present and prospective research work of IoT in E-Cardiology. This study also indicates how different communication technologies bring privacy and security to IoT devices related to cardiac healthcare. The purpose of this paper is to highlight the influence of IoT, communication technologies, and AI techniques in cardiac healthcare in light of our systematic literature review protocol. Therefore, one can say that this systematic review covers the complete and latest infrastructure of E-Cardiology, along with its benefits and challenges which have not been examined before in such a comprehensive way.

**Author Contributions:** Conceptualization, R.I. and M.A.K.; methodology, R.I. and U.U.; validation, U.U. and S.N.; formal analysis, U.U. and S.N.; investigation, U.U. and S.N.; resources, R.I., U.U. and S.N.; writing—original draft preparation, U.U., S.N. and A.U.; writing—review and editing, R.I., U.U. and S.N.; visualization, A.U. and S.N.; supervision, R.I. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research received no external funding.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Appendix A**

*Selected Literature*

Summary of the selected literature consisting of 104 papers used in this systematic literature review is given in the Table A1.

**Table A1.** Selected Literature.

#### **Table A1.** *Cont.*

#### **Table A1.** *Cont.*

#### **References**

## *Review* **Deep Learning for LiDAR Point Cloud Classification in Remote Sensing**

**Ahmed Diab 1, Rasha Kashef 2,\* and Ahmed Shaker 1,\***

**Abstract:** Point clouds are one of the most widely used data formats produced by depth sensors. There is a lot of research into feature extraction from unordered and irregular point cloud data. Deep learning in computer vision achieves great performance for data classification and segmentation of 3D data points as point clouds. Various research has been conducted on point clouds and remote sensing tasks using deep learning (DL) methods. However, there is a research gap in providing a road map of existing work, including limitations and challenges. This paper focuses on introducing the state-of-the-art DL models, categorized by the structure of the data they consume. The models' performance is collected, and results are provided for benchmarking on the most used datasets. Additionally, we summarize the current benchmark 3D datasets publicly available for DL training and testing. In our comparative study, we can conclude that convolutional neural networks (CNNs) achieve the best performance in various remote-sensing applications while being light-weighted models, namely Dynamic Graph CNN (DGCNN) and ConvPoint.

**Keywords:** point clouds; deep learning; remote sensing

#### **1. Introduction**

The light detection and ranging (LiDAR) mapping generate precise spatial information about the shape and surface components of the Earth. Advancements in LiDAR mapping systems and their technologies have been proven to examine natural and manmade environments across various scales with higher accuracy, precision, and flexibility [1]. LiDAR Remote sensing provides an accurate 3D representation of scanned areas with many features that provide great performance for various applications. Such applications include Digital Elevation Model (DEM), Digital Surface Model (DSM), and Digital Terrain Model (DTM) generation, which, combined with intensity data, achieve excellent performance in urban land cover classification [2]. Some other urban applications include pavement crack detection [3], collapsed building detection [4], road markings and fixtures extraction and classification [5], cultural heritage classification [6], and change detection [7]. Because LiDAR is sensitive to variations in vertical vegetation structure, it makes it very effective for natural resources [8] and forest applications [7], such as tree species classification [9]. Additionally, full-waveform LiDAR adds more advantages to using LiDAR in forestry applications [10].

Various deep learning models have been developed with outstanding performance for data classification on point cloud datasets in multiple applications. Existing deep learning methods for point cloud classifications involve architectures based on the traditional neural network, the Multi-Layer Perceptron (MLP). These models are called PointNet-Based as they build on the pioneering work of PointNet [11]. PointNet is a great performer that is very lightweight but suffers from local information loss. Global features are features of a scene, object, or image that describe it as a whole, compared to local features that are extracted at different points and represent patches of the scene or image [12]. PointNet++ [13]

**Citation:** Diab, A.; Kashef, R.; Shaker, A. Deep Learning for LiDAR Point Cloud Classification in Remote Sensing. *Sensors* **2022**, *22*, 7868. https://doi.org/10.3390/s22207868

Academic Editors: Andrei Velichko, Dmitry Korzun and Alexander Meigal

Received: 12 September 2022 Accepted: 9 October 2022 Published: 16 October 2022

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

mitigates the loss by building a feature aggregation pyramid to learn hierarchically, similar to how a traditional Convolutional network learns. One of the biggest challenges of using LiDAR point clouds in deep learning is the unstructured shapes of the point cloud data; a convolutional kernel that works on uniform grid-structured data cannot be directly applied to the raw point cloud. A convolutional neural network can better capture spatial features, which performs better than a traditional neural network while being more lightweight than most handcrafted models. The convolutional neural network is structured as a convolution layer, non-linearity, e.g., Rectified linear unit (ReLU), and pooling layers to distil features from low-level to high-level [14]. Applying CNNs on point clouds involves the 2D projection of the point cloud to obtain images that can then be fed into traditional convolution layers in a convolutional neural network. Another approach is resampling or restructuring the point cloud into uniform volumetric grids using occupancy functions and 3D convolutional layers to create the CNN or to design novel convolutional layers that can operate on pointsets and the custom convolution operation to build the CNN.

This paper provides a roadmap for current DL deep learning models for LiDAR point cloud classifications in remote sensing. Existing deep learning methods can be classified as projection-based and point-based models. Each category enjoys specific characteristics; however, they show some limitations. Thus, this paper summarizes the significant subcategories: 2D projection, Multiview projection, voxelization, Convolutional-based networks, and graph convolutional networks. Additionally, we cover some examples that encompass most of the fundamentals within each subcategory. Remote sensing applications require different datasets or workflows; thus, we cover some examples from remote sensing that employ or build upon computer vision models. Our comparative analysis shows that DGCNN and ConvPoint have shown the best performance in various remote-sensing applications while being light-weighted models. The rest of this paper can be organized as Section 2 focuses on LiDAR point cloud data and processing overview, Section 3 introduces the primary computer vision deep learning models that are often used to classify 3D data, and Section 4 presents Point cloud computing tasks that are common in remote sensing applications, Section 5 introduces the benchmark 3D datasets used in training and testing of deep learning models grouped as objects, indoor, arial scanned, mobile scanned, and terrestrial scanned datasets, Section 6 shows the evaluation metrics commonly used to measure and benchmark model performance; Section 7 provides a comparative analysis of existing models on different datasets for different classification tasks. Finally, Section 8 concludes the paper.

#### **2. LiDAR Point Clouds**

A typical LiDAR system in remote sensing uses a laser, Global Positioning System (GPS) and an Inertial Measurement Unit (IMU) to approximate the heights of objects on the ground. Discrete LiDAR data are generated; each point represents high energy points along with rebounded energy. Discrete LiDAR points contain each point's x, y, and z values. The z value is used to obtain height. The LiDAR data can estimate surface structures with various methods [15]. The raw LiDAR data are delivered as points, known as point clouds, that can be further processed to create Digital Elevation Models (DEMs) or Triangulated Irregular Networks (TINs) [1]. Point data are commonly stored in LAS (LASer) format, regarded as an industry standard that contains information in a binary file specific to the LiDAR nature of data without being complex [15]. The LiDAR data can also contain other information such as the intensity of the rebounds, the point classification (if applicable), number of returns, time, and source of each point [1,15]. LiDAR scanners use a laser pulse to measure the distance from the sensor using the time for the laser pulse to return in the case of time-of-flight sensors (Figure 1a) [16] or using the triangulation angle on the optical sensor for triangulation-based scanners (Figure 1b) [17]. The LiDAR scanners then generate an [x, y, z] position relative to the sensor's locations based on the distance from the sensor and the degrees of rotation of the sensor, such as pitch, roll, and yaw [18]. Most LiDAR sensors also measure the intensity of the return signal, which can be used to differentiate

between different surface types with different reflectivity [1]. Additionally, the sensor is often paired with a GPS and an IMU to capture data required for georeferencing and mapping of the point cloud.

**Figure 1.** Time of Flight LiDAR sensor calculation (**a**) [16] and triangulation-based LiDAR calculation (**b**) [17].

For supervised classification, a significant challenge when working on LiDAR point clouds is the variation in density inherent in the nature of the data. The density of similar objects is also varied, as it depends on the speed of the vehicle mounting the sensor. Some areas will be too dense and expensive to process, requiring some form of downsampling. Other regions of a point cloud will have few or no points present. Additionally, for LiDAR point clouds that include intensity values, the intensity of the same object could be affected by different conditions and result in the same object having slightly different intensities [18].

#### **3. Point Cloud Computing**

Remote sensing data go through multiple processing steps to generate information that can be consumed for production. Over the past few years, deep learning has been applied to almost all remote sensing data processing aspects. Most notably, classification and segmentation tasks. Regarding remote sensing 3D LiDAR point clouds, there is limited interest in whole scene classification and more in semantic classification or segmentation tasks. Some other examples of deep learning tasks tackled by deep learning include change detection, registration, fusion, and completion.

Traditionally, deep learning classification describes classifying an entire scene or an object as belonging to a specific class as a whole. One example of classification tasks that use 3D point clouds in remote sensing is the classification of tree species or roof types previously segmented. However, remote sensing classification tasks involve semantic classification and segmentation rather than aiming to identify an entire scene or object to a single class. A significant example of semantic classification is Land use/Land cover classification of Terrestrial and Arial Laser scanned (TLS/ALS) data. Segmentation divides and assigns the data into different target classes and is split into three types, semantic, instance, and panoptic segmentation [19]. Semantic segmentation assigns every point/pixel from the input data to one of the target classes without distinguishing different objects; for example, all tree points will be labelled trees. Instance segmentation involves identifying and labelling objects belonging to target classes while distinguishing them from each other, such as tree1, tree2, etc. Panoptic segmentation classifies every point/pixel in the input as part of a class while distinguishing separate objects of a class from each other [19].

The most common application of image fusion in LiDAR remote sensing is the fusion of 3D point clouds and RGB images to train a deep learning model for classification and segmentation tasks [20–22]. The features extracted from both types of data are used to enhance the performance of each class in the application of each class. Registration is the process of matching and aligning two or more images or point clouds in the case of LiDAR data obtained from different viewpoints and/or using different sensors; one example is illustrated in [23], which achieves state-of-the-art performance. Completion is the process of filling in missing information from datasets that could result from the limitations of the sensors, conditions at the time of data capture, or the method of capture. For far-away distances, the spatial resolution of a LiDAR sensor is lower, sometimes resulting in finer details, such as road markings, signs, poles, etc., showing up incomplete. One example of completion can be found in [5]. Most completion tasks on LiDAR point clouds are done before training a classification model to improve performance and robustness.

#### **4. Deep Learning Models**

Advances have been made to produce DL models that are lightweight and efficient. Feature learning models on 3D point clouds can be categorized as projection-based and point-based models. This section briefly discusses models used as backbones or improved for newer networks.

#### *4.1. Projection-Based Methods*

Some projection-based models create 2D projections from 3D point clouds and use traditional 2D feature learning. This process primarily depends on projection direction (X, Y or Z—default: Z) and other aspects such as the grid (size, scale, shape). Other projection models create volumetric grids or voxels through 3D feature extraction layers.

• 2D Convolutional Neural Networks

U-Net [24]: builds on a fully convolutional model and extends it to work with few training data while providing better performance. The U-Net architecture consists of repeated two unpadded 3 × 3 convolutions followed by ReLU and downsampling 2 × 2 max pooling with stride 2. For each convolution step, the number of feature channels is doubled. In the deconvolution steps, the features are upsampled and followed by a 2 × 2 convolution that halves the number of channels. The resulting feature map goes through cropping and two 3 × 3 convolutions followed by a ReLU. The cropping is necessary because of the border pixels lost after every convolution. Finally, a 1 × 1 convolution is applied to label pixels and generate segmentation results.

DeepLab [25]: employs atrous convolution [25,26] to change the scope of convolution and extract global features while also allowing larger networks without extra parameters. DeepLab proposes Atrous Spatial Pyramid Pooling (ASPP) to segment at different scales

by applying the same filters at different sampling rates and field-of-views, then the outputs are added together. To overcome the toll downsampling and max pooling operations in deep convolutional neural networks (DCNNs), DeepLab implements the fully connected Conditional Random Field (CRF) from [27], which is trained separately from the rest of the network. Iterations DeepLabV3 [28] and DeepLabV3+ [29] improve the performance of DeepLab. Unlike [25], DeepLabV3 [28] performs batch normalization within ASPP. Additionally, global average pooling is applied to the last feature map. The resulting imagelevel features are fed into a 1 × 1 convolution with 256 filters, then multiplied to the desired spatial dimension. DeepLabV3 abandons the CRF and replaces it with concatenating and aggregating the resulting features and passing them through another 1 × 1 convolution with 256 filters before computing the final logits. DeepLabV3+ [29] uses a decoder module to refine segmentation results, especially around object boundaries. Depth-wise separable convolutions are applied to ASPP pooling and decoder modules resulting in a faster and more robust network.

VGGNet [30] evaluates the effect of increasing the network depth of a convolutional network using very small 3 × 3 convolution filters. It improves the classification performance compared to previous state-of-the-art models by pushing the depth to 16–19 weight layers. ResNet [31] adopts residual learning to every stacked layer in the convolutional network. The shortcut connections are added without increasing parameter or computation complexity. The residual learning allows deep networks with performance gain over shallower networks.

• Multiview representation

MVCNN [32] tackles 3D feature learning using traditional image-focused networks by making 2D renders of the 3D object from different angles and passing it through a standard CNN. MVCNN generates 80 views of the 3D object by placing 20 virtual "cameras" pointed at the object's centroid, then generates 4 renders per camera at 0-, 90-, 180-, and 270-degree rotation along the axis through the camera and object center. After each image is passed through the first CNN, the outputs are aggregated at a view-pooling layer which performs element-wise maximum operation across the different input views before passing through the remaining section of the network, i.e., the second CNN.

• Volumetric grid representation

VoxNet [33] uses occupancy grids to efficiently estimate occupied, free, and unknown space provided by ranging measurements. Small (32 × 32 × 32 voxels) dense voxels are used to optimize GPU usage. VoxNet uses a more basic 3D CNN to extract and learn features, consisting of 5 of two convolution layers, a convolution and pooling layer, and two fully connected layers. The model can perform object classification in real-time while achieving state-of-the-art performance. VoxelNet [34] introduces a multi-layer voxel feature encoding (VFE) that enables inter-point interaction within a voxel. The point cloud is divided into equally spaced voxels encoded using the stacked VFE layers, allowing complex local 3D information learning. VoxelNet works on object detection using a Region Proposal Network (RPN) at the final stage to create bounding boxes.

#### *4.2. Point-Based Methods*

Point-based methods consume unstructured and unordered point clouds. Some of the models covered in this section are used as backbones or parts of a larger architecture, while others are adapted for remote sensing tasks with minimal modifications.

• PointNets

PointNet [11] directly consumes point cloud data for feature extraction. The network provides a unified approach to 3D recognition that can be applied for various tasks such as object classification, instance segmentation, and semantic segmentation. PointNet uses Multi-Layer Perceptrons (MLPs) combined with a joint alignment network. To hold invariance under geometric transformations, the input is passed through a T-Net module [11], where it is multiplied by an affine transformation matrix. PointNet provides great performance while remaining lightweight and computationally efficient. PointNet cannot produce local features of neighbouring points; PointNet++ [13] introduces a class pyramid feature aggregation scheme. The scheme comprises three stacked layers: the sampling layer, the grouping layer, and the PointNet layer. This allows PointNet++ to extract features in a hierarchical fashion similar to traditional image learning, reducing local information loss. PointASNL [35] is an end-to-end network that effectively deals with noisy point clouds. The two primary components of the model are the adaptive sampling (AS) and the localnonlocal (L-NL) modules. Initially, the AS module reweighs neighbour points surrounding the initial sampled points from the farthest point sampling and then adaptively adjusts the sampled points beyond the point cloud. The L-NL module captures the neighbour and long-range dependencies of the sampled point. Self-Organizing Network (SO-Net) [36] generates a Self-Organizing Map (SOM) to simulate point cloud spatial distribution. The SOM retrieves hierarchical features from individual points and SOM nodes. A Point-tonode search is performed on the output of the SOM for each point. Each point is normalized, and features are learned through a series of fully connected layers. Node feature extraction is done through channel-wise max-pooling the point features. Final learned features are extracted using a batch of fully connected layers referred to as a small PointNet.

### • (Graph) Convolutional Point Networks

ConvPoint [37] proposes continuous convolution kernels to allow arbitrary point cloud sizes. Points {q} are selected iteratively from the input point cloud {p} until the target number of points is reached through a score-based process. Using a kd-tree built on the input point cloud, K-nearest neighbour search from {p} is performed on points in {q}. A convolution operation is performed for each subset, generating the output features. Operations detailed by ConvPoint are successfully adapted for classification, part segmentation, and semantic segmentation tasks. ConvPoint can produce significant performance with time- and cost-efficient. Dynamic Graph CNN (DGCNN) [38] generates local neighbourhood graphs and applies convolution on the edges connecting neighbour point pairs. Unlike traditional graph CNNs, DGCNN uses a dynamic graph where the set of k-nearest neighbours for a point change between layers in the network and is calculated from the sequence of embeddings. The EdgeConv block introduced by DGCNN computes edge features for each input point and applies an MLP followed by channel-wise symmetric aggregation. Taylor Gaussian mixture model (GMM) network (TGNet) [39] is composed of units named TGConv that perform convolution operations parametrized by a family of filters on irregular point sets. The filters are products of geometric features expressed by Gaussian weighted Taylor kernels and local point features extracted from local coordinates. TGConv features are aggregated using parametric pooling to generate feature vectors for each point. TGNet uses a CRF at the output layer to improve segmentation results.

#### **5. Benchmark Datasets**

Advancements in Deep learning on point clouds have attracted more and more attention, especially in the last few years. Several publicly available datasets were also released, which helped further support research on DL development. An increasing number of methods have been introduced to deal with various challenges related to point cloud processing, including 3D shape classification, 3D object detection and tracking, 3D point cloud segmentation, 3D point cloud registration, 6-DOF pose estimation, and 3D reconstruction [18]. Table 1 briefly overviews some of the most commonly used publicly available point cloud datasets. Outdoor datasets are classified based on acquisition technique, Aerial, Mobile, or Terrestrial Laser scanned data or ALS, MLS, and TLS, respectively. The remaining datasets in this paper are indoor laser-scanned datasets and datasets of object scans. While ModelNet40 and S3DIS are not LiDAR scanned datasets, they are included as we found that they are the most commonly tested datasets for their respective tasks in remote sensing classifications. ModelNet40 dataset consists of CAD files; most point cloud network testing uses a point cloud sampled from the 3D object files. The models that used the ModelNet40

dataset outlined later in the paper are tested on the dataset by sampling the objects into a point cloud and then applying the model. Similarly, S3DIS, while not LiDAR data, is a point cloud and the models tested on it are suitable for point clouds obtained from LiDAR scans.

**Table 1.** Benchmark datasets for training and testing deep learning on 3D point clouds.

#### **6. Performance Metrics**

Various evaluation metrics have been used for segmentation, detection, and classification. The summary of the evaluation metrics [53] is shown in Table 2. Metrics for segmentation, detection, and classification are the intersection over union (IoU), mean IoU, and overall accuracy (OA) [53]. Detection and classification results are mainly analyzed using precision, recall and F1-score, which takes the true positives (TP), false positives (FP), and false negatives (FN) for calculation.

**Table 2.** Performance Evaluation Metrics.

#### **7. Comparative Analysis**

The datasets ModelNet40, S3DIS, and Toronto3D provide an overview of benchmarks used for different classification tasks: object classification, indoor scene classification, and urban outdoor classification. Table 3 shows the performance comparison for the current 3D object classification, indoor scene segmentation, and outdoor urban semantic segmentation models using various evaluation metrics. The best-performing configuration for each model was selected. For example, using a higher sampled point cloud in ModelNet40 tests can produce better performance. Therefore, if the authors tested the models using different point counts, the best set of results is used. The results outlined in the table are obtained from the testing by each model's respective author(s) except for the ConvPoint results on Toronto3D, which we tested for this paper. From Table 3, we can see that DGCNN and ConvPoint achieve the best performance on most datasets while being lightweight relative to models with similar performance. Additionally, these two models have been tested on multiple different tasks and different types of datasets. The major limitation of ConvPoint is that the convolutional layer introduced is a scale agnostic, i.e., the object's size is important for scans and provides valuable information. DGCNN could be further improved by adjusting the implementation details to improve the computational efficiency of the model.

Most remote sensing papers use one of the previously outlined computer vision models. The model is deployed directly for the application dataset or modified and attached to post and/or preprocess pipelines. To further test the performance of the ConvPoint model in this paper, we have also experimentally trained ConvPoint on Toronto3D using labels such as L001, L003, and L004 and used L002 for testing. The training was run using batch size 8, block size 8, and #of points 8192 for 100 Epochs. The testing results are marked with a (\*) in Table 4. Table 4 includes some applications categorized according to their dataset, performance, and remote sensing deployment. We can conclude that both DGCNN and ConvPoint have shown promising results across the different applications in remote sensing.

#### **Table 3.** Comparative Analysis of Deployed Models.

**Table 4.** Overview of some deep learning contributions focused on remote sensing data.

#### **Table 4.** *Cont.*

<sup>1</sup> f1-Score is denoted by \*, mIOU is denoted by ˆ and OA is denoted by ~.

#### **8. Conclusions and Future Directions**

Recent work on the advances of deep learning on LiDAR 3D point cloud processing was analyzed and summarized. An overview of the different model types and the stateof-the-art and/or fundamental models of each type was provided. Additionally, the performance of the models was provided on datasets for different classification tasks. The strongest performing models were trending towards 3D Graph CNNs and 3D CNNs [69,70] that work directly on the raw point cloud data. These models can provide state-of-the-art performance and remain computationally lightweight. Finally, different applications of remote sensing that deploy deep learning models were overviewed. One major challenge when comparing the remote sensing models was the lack of standardized test datasets and the frequent use of proprietary datasets. Notable test datasets available are Toronto3D, Paris-Lille 3D, ISPRS 3D, and S3dIS. Future Directions would involve expanding the application of the state-of-the-art methods in autonomous driving [71,72].

**Author Contributions:** Conceptualization, A.D., R.K. and A.S.; methodology, A.D., R.K. and A.S.; software, A.D., R.K. and A.S.; validation, A.D., R.K. and A.S.; formal analysis, A.D., R.K. and A.S.; investigation, A.D., R.K. and A.S.; resources, A.D., R.K. and A.S.; data curation, A.D., R.K. and A.S.; writing—original draft preparation, A.D., R.K. and A.S.; writing—review and editing, A.D., R.K. and A.S.; visualization, A.D., R.K. and A.S.; supervision, R.K. and A.S.; project administration, R.K. and A.S.; funding acquisition, R.K. and A.S.; All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the Natural Sciences and Engineering Research Council of Canada (NSERC), grant number [RGPIN-2020-05857], and Smart Campus Integrated Platform Development Alliance project with FuseForward. The APC was funded by Toronto Metropolitan University.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Not applicable.

**Acknowledgments:** We acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC), [funding reference number RGPIN-2020-05857], and Smart Campus Integrated Platform Development Alliance project with FuseForward.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

MDPI St. Alban-Anlage 66 4052 Basel Switzerland www.mdpi.com

*Sensors* Editorial Office E-mail<EMAIL_ADDRESS>www.mdpi.com/journal/sensors

Disclaimer/Publisher's Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Academic Open Access Publishing

www.mdpi.com ISBN 978-3-0365-8429-4