**Figure 2.** Measured pump flow signals were used to calculate *SI*, whose values gradually decreased and finally approached the setpoints of 9 for AFP (**a**) and 5 for DP2 (**b**), respectively.

**Table 2.** Performance comparison among the proposed SIMPF control strategy and other control algorithms during various test conditions with DP2.

1 With 5% noise. 2 With 10% noise.

**Figure 3.** Comparison of performance among six control strategies at rest when the PVR increase eight times for AFP. The increase in PVR started when *t* = 300 s. (**a**−**c**) SIMPF control with 5% noise. (**d**−**f**) Δ*RPM* control. (**g**−**i**) CPS control. (**j**−**l**) Δ*P* control. (**m**−**o**) MAoP control. (**p**−**r**) LVEDP control. SIMPF and LVEDP control did not cause any LV suction. Δ*RPM*, CPS, and Δ*P* control induced intermittent suction. MAoP control induced constant suction.

**Figure 4.** Comparison of performance among six control strategies under step change from exercise to rest for AFP. The transition started when *t* = 300 s. (**a**−**c**) SIMPF control with 5% noise. (**d**−**f**) Δ*RPM* control. (**g**−**i**) CPS control. (**j**−**l**) Δ*P* control. (**m**−**o**) MAoP control. (**p**−**r**) LVEDP control. Intermittent suction events were found for the MAoP control algorithm.

**Figure 5.** Comparison of performance among six control strategies at rest when the PVR increase eight times for DP2. The increase in PVR started when *t* = 300 s. (**a**−**c**) SIMPF control with 5% noise. (**d**−**f**) Δ*RPM* control. (**g**−**i**) CPS control. (**j**−**l**) Δ*P* control. (**m**−**o**) MAoP control. (**p**−**r**) LVEDP control. SIMPF and LVEDP control did not cause any LV suction. Δ*RPM* and Δ*P* control induced intermittent suction. CPS and MAoP control induced constant suction.

**Figure 6.** Comparison of performance among six control strategies under step change from exercise to rest for DP2. The transition started when *t* = 300 s. (**a**−**c**) SIMPF control with 5% noise. (**d**−**f**) Δ*RPM* control. (**g**−**i**) CPS control. (**j**−**l**) Δ*P* control. (**m**−**o**) MAoP control. (**p**−**r**) LVEDP control. Intermittent suction events were found for the Δ*RPM* and MAoP control algorithms.

#### *3.2.* Δ*RPM Control Algorithm*

The Δ*RPM* control algorithm successfully generated sufficient cardiac outputs and avoided LV suction events during rest and exercise conditions for both pumps (Tables 1 and 2). However, as shown in Figures 3d–f and 5d–f, an intermittent LV suction event occurred when the PVR was increased under rest, since the instantaneous LV pressure decreased to less than 1 mmHg. Meanwhile, no LV suction was found under exercise when the PVR was increased. The control algorithm could function effectively during the rapid step change from rest to exercise (no figures shown) and exercise to rest (Figure 4d–f) for AFP. However, for DP2, there was a serious intermittent LV suction event (~100 s), causing the minimum LVP to be −7.1 mmHg during the rapid transition from exercise to rest (Figure 6d–f).

#### *3.3. CPS Control Algorithm*

The CPS control strategy did not induce LV suction events without a change in PVR and provided adequate physiological perfusion during the rest condition (5.0 L/min) for both pumps. However, during exercise, the increase in pump flow was lower than that using any other control algorithm, especially for DP2 (Tables 1 and 2). Figure 3g–i also shows that this control algorithm caused intermittent LV suction at rest with AFP when the PVR was octupled compared to the normal activities of the patients. It caused constant suction with DP2, as shown in Figure 5g–i. Furthermore, the CPS control strategy did not trigger LV suction events during rapid condition change from rest to exercise (no figure shown) and exercise to rest (Figures 4g–i and 6g–i) and for both pumps.

#### *3.4.* Δ*P Control Algorithm*

Maintaining a fixed ΔP from LV to the aorta across AFP or DP2 provided physiological demands of 5.0 L/min and 8.7 L/min during rest and exercise conditions, respectively (Tables 1 and 2). However, intermittent suction events were observed at rest and during exercise with a rapid 8-fold increase in PVR (Figures 3j–l and 5j–i). No suction events were found during transitions between rest and exercise conditions.

#### *3.5. MAoP Control Algorithm*

For both pumps, the MAoP control algorithm guaranteed adequate end-organ perfusion at rest (5.0 L/min) and exercise (9.0 L/min) (Tables 1 and 2). However, MAoP control failed to adapt to sufficient end-organ perfusion under rest (Figures 3m–o and 5m–o) and exercise states when the PVR increased eight times, because the onset of constant suction was observed as the minimum value of LVP was negative. The controller also caused intermittent suction cases during the transition from exercise to rest (Figures 4m–o and 6m–o).

#### *3.6. LVEDP Control Algorithm*

The LVEDP control strategy increased the pump flow rate from 5.0 L/min to 8.5 L/min for both pumps when the physiologic condition changed from rest to exercise (Tables 1 and 2). No suction events were observed during all the tested conditions (Figure 3p–r, Figure 4p–r, Figures 5p–r and 6p–r).

In summary, based on all the simulation results, the proposed control algorithm is pump-independent and outperformed other control strategies for both axial and mixed flow pumps by avoiding suction and providing physiologic levels of perfusion.

#### **4. Discussion**

The computer simulation results demonstrated the feasibility and performance of the proposed SIMPF control strategy to autonomously regulate pump flow rates. The human circulatory system has highly non-linear dynamics including flow discontinuities due to the presence of valves, and highly variable physiological perfusion needs. Similarly, the LVAD pump dynamics are non-linear and depend on the design of the pump. LVAD support to the circulatory system has several requirements and constraints, which makes the design of a control algorithm highly challenging: (1) LVADs have low preload and

afterload sensitivity; (2) appropriate amounts of flow to meet varying cardiac demand must be maintained for the functional capacity of patients; and (3) over-pumping and suction need to be avoided even during the rapid reduction in preload (e.g., Valsalva). The proposed SIMPF algorithm adequately met the conflicting demands of maintaining perfusion demand and avoided LV suction events under various simulated conditions. Notably, the SIMPF control algorithm was effective at avoiding suction even when PVR was increased resulting in a drastic reduction in preload. Similarly, quick step transitions between exercise and rest were also simulated to produce conditions that are conducive to causing suction events. The proposed SIMPF control algorithm measured pump flow with 5–10 percent uniformly distributed noise added to the PF signals, which were used to extract *SI*. Up to 10% noise was used as this is the maximum error for ultrasonic flow probes used clinically.

Axial flow and mixed flow pumps are two different types of pumps. Axial pumps are based on the Archimedes screw, where the flow of the liquid is along the axis of the impeller. Mixed flow is a centrifugal flow pump with a mixed flow impeller. In the DP2 pump, the fluid experiences both radial acceleration and lift, and exits the impeller nearly perpendicular to the axial direction. There are differences in pump dynamics between the axial flow pump and DP2 pump, and different sensitivities between the two devices to the pressure head (HQ curves). However, the proposed SIMPF algorithm was able to achieve similar results using both AFP and DP2, demonstrating the pumpindependence of the proposed control method. The *SI* values between the two pumps were notably different, as expected due to their intrinsic differences causing differences in pump flow used to extract *SI*, and also because pump types are affected differently by the contractility of the native heart. The SIMPF algorithm requires the direct measurement of flow. Measuring LVEDP using pressure sensors can provide similar performance compared to the SIMPF control strategy. However, pressure measurements are prone to failure and long-term drift due to contact with blood. A sensor drift of even 2–3 mmHg can cause significant performance degradation and suction with the LVEDP control algorithm. Unlike pressure sensors, ultrasonic flow probes have been successfully used for long term LVAD flow measurements. When mounted on the outflow graft of an LVAD, the performance of ultrasonic flow probes is not significantly affected by tissue ingrowth. Unfortunately, flow probe measurement noise is unavoidable, and control algorithm performance degrades with increasing measurement noise. However, even with 10% noise, the algorithm prevented suction and provided physiologic perfusion. The chosen measurement noise levels (5–10%) were based on the levels reported for ultrasound flow probes. The performance of the control algorithm can be further improved by periodic calibration of the flow probe.

The SIMPF algorithm requires some level of native LV function to generate the variation in flow rate for *SI*. This native ventricular contractility is always present clinically. If ventricular asystole occurs, it would also result in the loss of right ventricular function and lead to mortality, even in the presence of adequate LVAD support. During exercise, an increase in preload due to venous return will increase native ventricular contractility due to the Frank–Starling mechanism. This increase in contractility will increase *SI*, which in turn will result in increased LVAD flow to meet the perfusion demand. Similarly, a reduction in LV contractility caused by reduced physiological demand would generate lower pump flow rates to prevent LV suction. A 5 s moving window was used in this study to minimize oscillations in *SI* values due to varying heart rates that are independent of physiological conditions. A shorter moving window will increase the speed of response, but may be more sensitive to noise and transient events.

The performance of the proposed SIMPF control strategy was superior compared to previously proposed control algorithms. It provided physiologically relevant cardiac outputs that were similar to other algorithms proposed in the literature, but was better at avoiding suction compared to the other sensor-based and sensorless control algorithms, especially when elevated PVR occurred briefly during the Valsalva maneuver or coughing. Rapid and sustained PVR increase by eight-fold during rest and exercise does not occur in nature. This was done to demonstrate the robustness of the controller to avoid suction even under non-physiologic, extreme conditions. Furthermore, sensor based control algorithms are usually model-based and predict pressure heads and flows. They usually require *a priori* knowledge of the blood viscosity or can be erroneous in the case of inflow or outflow kinking or thrombus formation in the blood pump. While incorporating a flow sensor increases complexity, an actual measurement of flow obviates the need for estimation and improves the performance of the controller under inflow/outflow graft kinks or thrombus formation that occurs in patients implanted with LVAD. The flow probes that were previously implanted with DeBakey/Heart Assist 5 LVADs underwent rigorous durability testing prior to approval by the Food and Drug Administration and no flow probe failures have been reported during LVAD implants.

In this study, the in silico computer simulation model provided a meaningful initial step for early tested hypotheses, but it cannot replace mock flow loop studies, animal testing, and clinical trials. For instance, it cannot fully replicate the complex in vivo dynamics including tissue remodeling, autonomous regulation, and neurohumoral responses. The lumped parameter human circulatory system model has several inherent limitations due to assumptions of ideal heart valves, Newtonian blood, and ignores the effects of gravity and inertia. However, the in silico model demonstrated the feasibility control algorithms despite these limitations. Pre-clinical in vitro and in vivo experiments will be used to validate the SIMPF control algorithm.

#### **5. Conclusions**

A new flow sensor-based SIMPF control strategy was developed for rotary LVADs to provide adequate cardiac output and prevent LV suction. This proposed control strategy implemented the control objective using an effective *SI* extracted from the measured pump flow signal. Two different types of rotary LVADs were incorporated to quantify the performance of the SIMPF strategy, showing promising results. This algorithm is pump-independent and can be incorporated into existing LVAD control systems.

**Author Contributions:** Software, writing—original draft preparation, L.L.; Supervision, writing review and editing, K.Q.; Formal analysis, validation, A.S.E.-B., T.J.R., and P.S.; Funding acquisition, writing—review and editing, G.A.G.; Conceptualization, funding acquisition, methodology, writing review and editing, Y.W. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the LiaoNing Revitalization Talent Program of China (grant number XLYC1807016), the National Natural Science Foundation of China (grant number 32071314), and the National Institutes of Health R01 Grant (grant number R01HL151663).

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** Data are made available through the corresponding author upon a reasonable request.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

## *Article* **Atrial Fibrillation Classification with Smart Wearables Using Short-Term Heart Rate Variability and Deep Convolutional Neural Networks**

**Jayroop Ramesh, Zahra Solatidehkordi, Raafat Aburukba \* and Assim Sagahyroon**

Department of Computer Science and Engineering, American University of Sharjah, Sharjah P.O. Box 26666, United Arab Emirates<EMAIL_ADDRESS>(J.R<EMAIL_ADDRESS>(Z.S<EMAIL_ADDRESS>(A.S.)

**\*** Correspondence<EMAIL_ADDRESS>Tel.: +971-6-5152956

**Abstract:** Atrial fibrillation (AF) is a type of cardiac arrhythmia affecting millions of people every year. This disease increases the likelihood of strokes, heart failure, and even death. While dedicated medical-grade electrocardiogram (ECG) devices can enable gold-standard analysis, these devices are expensive and require clinical settings. Recent advances in the capabilities of general-purpose smartphones and wearable technology equipped with photoplethysmography (PPG) sensors increase diagnostic accessibility for most populations. This work aims to develop a single model that can generalize AF classification across the modalities of ECG and PPG with a unified knowledge representation. This is enabled by approximating the transformation of signals obtained from low-cost wearable PPG sensors in terms of Pulse Rate Variability (PRV) to temporal Heart Rate Variability (HRV) features extracted from medical-grade ECG. This paper proposes a one-dimensional deep convolutional neural network that uses HRV-derived features for classifying 30-s heart rhythms as normal sinus rhythm or atrial fibrillation from both ECG and PPG-based sensors. The model is trained with three MIT-BIH ECG databases and is assessed on a dataset of unseen PPG signals acquired from wrist-worn wearable devices through transfer learning. The model achieved the aggregate binary classification performance measures of accuracy: 95.50%, sensitivity: 94.50%, and specificity: 96.00% across a five-fold cross-validation strategy on the ECG datasets. It also achieved 95.10% accuracy, 94.60% sensitivity, 95.20% specificity on an unseen PPG dataset. The results show considerable promise towards seamless adaptation of gold-standard ECG trained models for non-ambulatory AF detection with consumer wearable devices through HRV-based knowledge transfer.

**Keywords:** biomedical informatics; cardiovascular disease; deep learning; ECG; heart rate variability; machine learning; PPG; smartphones; smart wearables

#### **1. Introduction**

Cardiovascular diseases (CVD) are the leading cause of death worldwide, with the World Health Organization (WHO) in 2016 estimated 17.9 million deaths annually [1]. CVD is a group of conditions that affect the heart's rhythm mechanical function, and electrical activity [2]. This is associated with an increased likelihood of strokes and heart failure. Timely detection through regular monitoring of CVD is necessary to improve the treatment process for heart conditions and lower the risk of mortality [3]. Cardiac arrhythmia is categorized under CVD and is characterized by the disordered electrical activity of the heart. An arrhythmia can manifest as irregularly rapid heart rhythms (tachycardia) or anomalous slow heart rhythms (bradycardia). AF is one of the most common types of cardiac arrhythmia. In this work, the focus is on the classification of (i) normal sinus rhythm (NSR), and (ii) atrial fibrillation (AF). Goldberger et al. [4] defines NSR as a rhythm with normal (1:1) atrioventricular conduction and a normal PR interval (the interval between atrial depolarization and ventricular depolarization) at a

**Citation:** Ramesh, J.; Solatidehkordi, Z.; Aburukba, R.; Sagahyroon, A. Atrial Fibrillation Classification with Smart Wearables Using Short-Term Heart Rate Variability and Deep Convolutional Neural Networks. *Sensors* **2021**, *21*, 7233. https:// doi.org/10.3390/s21217233

Academic Editors: Ayman El-baz, Guruprasad A. Giridharan, Ahmed Shalaby, Ali H. Mahmoud and Mohammed Ghazal

Received: 3 September 2021 Accepted: 26 October 2021 Published: 30 October 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

heart rate between 60 and 100 beats/min, although normal heart rates may vary between individuals. The work reported in [5] defines AF as an arrhythmia with uncoordinated atrial activation and characteristics of irregular beat-to-beat intervals, absence of repeating P waves (indicates atrial depolarization), and irregular atrial activity.

The common technique for the clinical diagnosis of cardiac arrhythmia is based on the electrocardiogram (ECG). The ECG is a test that uses skin level electrodes with built-in sensors to measure the heart's electrical activity and identify abnormal heart rhythms and additional pathological conditions [6]. However, despite the multi-faceted diagnostic nature of ECG, most dedicated ECG devices available currently are expensive and are typically used within clinical or limited ambulatory settings [7]. While wearable ECG devices are emerging commercially, individuals gravitate towards smart wearables that can serve general functions and are not only intended for health monitoring. Off-theshelf smartphones and wearable devices that use photoplethysmography (PPG) sensors can serve as an affordable alternative to existing ECG devices, albeit as a supplementary approach for screening and not for conclusive diagnosis. PPG sensors are optical light sensors that record blood volume variations at sensitive peripheral sites of the human body, such as fingertips, wrist, and earlobes [8]. Moreover, PPG sensors are currently used extensively by fitness tracking applications to estimate the physiological events of heart rate and heart rate variability (HRV) [9]. PPG signals differ morphologically from ECG signals but exhibit similar characteristics as the HRV. This is termed as pulse rate variability (PRV). The advantages of PPG sensor-based consumer devices for cardiac arrhythmia monitoring are that they are relatively less obtrusive than their ECG counterparts. Their ubiquitous nature facilitates higher adoption by the general population. Despite these advantages, PPG recordings are more susceptible to noise saturation and variations in signal quality caused by user movement and skin tones [10].

The data features extracted from ECG and PPG heart signals to develop learning algorithms can be categorized as temporal or morphological [11]. Temporal features are the time-domain metrics such as the time between heartbeats. Many deep learning works in this area pursue the development of morphology-based models using the PPG segments or corresponding images to leverage the robustness and have generally superior performance in classification problems. However, there are significant challenges in developing PPG-based analytical models due to the limited public availability of universally reviewed benchmark databases, as opposed to the abundant ECG signals databases. Moreover, signal quality and noise saturation can corrupt the performance of the developed models. The manual annotation process for creating labeled datasets is complex and has the consistency issue of interrater variability [3]. Interrater variability arises when multiple expert annotators are involved in labeling heart rhythms manually. Different labels are assigned to the same data instance due to differences in their specific experiences. In practice, it is difficult to reach an agreement across multiple experts if the data are not ideally preprocessed and motion artifacts are not eliminated. This is the case with the PPG signal annotation efforts in most of the literature. Moreover, most developed algorithms in the literature are only applicable in controlled clinical settings, which hinders early prognosis accessibility to the general population.

Although there are inherent morphological differences between ECG and PPG-based signals, the studies reported in [12,13] have exhibited a high degree of correlation between the signals, especially their corresponding temporal HRV features. HRV measures the variation in terms of time between consecutive instantaneous heartbeats, measured through the ECG [9].

The PRV and HRV parameters, derived from ECG and PPG, respectively, exhibit similar properties under certain conditions. The properties have higher levels of agreement/equivalence when the PPG signals are not excessively situated with motion artifacts. Various predictive and detection models have been implemented using different HRV metrics with standard statistical and machine learning approaches [14–19]. However, there are considerably fewer deep learning-based models oriented towards usage in smartphones

and wearable devices. Deep learning has recently emerged as an effective methodology for cardiac classification tasks [20]. The experiments reported in [21–24] have achieved successful ECG signal classification using ECG databases by implementing convolutional neural networks. However, the existing approaches are designed for use in controlled hospital settings.

This research addresses the scarcity of publicly available PPG datasets, limited reproducible approaches in the existing literature, and varying sensor specifications. This work proposes implementing a deep learning approach that utilizes the knowledge transfer paradigm for cross-domain generalizability by training a model on ECG databases and adapting the developed model for PPG signals-based AF classification. The commonality in the distribution of temporal features derived from HRV (ECG) and PRV (PPG) is leveraged as input features to implement a one-dimensional convolutional neural network for classifying NSR and AF rhythms. The motivation for this approach is to introduce generalizable deep learning models that can mitigate the challenges associated with purely PPG- based analytical models and facilitate close to real-time AF detection.

The contributions of this work are as follows:

This paper is organized as follows: Section 2 introduces the background of the concepts used in the analysis of this work, Section 3 details the proposed approach, Section 4 presents the obtained results of the model, Section 5 discusses the results, and is followed by the conclusion and future work in Section 6.

#### **2. Background**

#### *2.1. Heart Activity Measures*

The entire sequence of a single heartbeat, beginning with the initial atrial excitation and concluding with the exit from the ventricular chambers, is called PQRST and is shown in Figure 1. An electrical impulse travels through the heart during each heartbeat, causing the heart muscles to pump blood. After a flat line driven by the impulse traveling to the bottom heart chambers, the right and left atria (upper heart chambers) create the first wave, called P wave. The right and left ventricles (bottom chambers) make the next wave called the QRS complex, and the final T wave indicates the repolarization of the ventricles. The QRS complex is the peak shown in Figure 1. Variations in parameters obtained from ECG and PPG, such as the duration and rate of heartbeats, can help detect abnormal heart activity [6].

PPG is an optical light-based technique to measure the volumetric change of the heart. As the heart contracts, blood pressure in the left ventricle (bottom chambers) increases. This is reflected by an increased pressurized pulse of blood into the capillaries and arteries of the body, indicated by discoloration of the skin. An LED light measures the difference in the amount of light reflected from sensitive areas, where the arteries are close to the skin, such as fingertips or earlobes, which is then used to measure an individual's heart rate [25]. A typical waveform of the PPG signal and its characteristic parameters are shown in Figure 2, which are the systolic peak, pulse with and diastolic peak, and dicrotic notch. Smartphones and wearable devices are generally accurate in acquiring PPG signals when the user is at rest, but potential inaccuracies are introduced because of motion artifacts and diverse skin tones. Motion artifacts typically occur due to misplacement of sensors such

that it does not make sufficient contact with the measurement site. Various skin tones affect the reflective properties of the optical light differently and therefore affect the accurate assessing of the changes in blood volume under the skin [26].

**Figure 1.** Single heartbeat sample with the QRS complex [4].

PPG has two peaks corresponding to the blood volume changes in the microvascular bed of tissue around the physical measurement site of the fingertips, earlobes, wrists, etc. Systolic peak is caused by the direct pressure wave traveling from the left ventricle to the body periphery (heart contraction). The diastolic peak reflects the pressure wave by arteries in the lower body (heart relaxation). The pulse width correlates with systemic vascular resistance, and the dicrotic notch reflects a transient increase in aortic pressure [27]. Although PPG is an indirect way to record the heart's activity, it has a high correlation with ECG signals. Its portability and relatively inexpensiveness make it a valuable alternative method to monitor cardiac activity [8].

#### *2.2. Heartrate Variability*

The HRV phenomenon is controlled by the Autonomous Nervous System (ANS) and is a direct result of the behavior of the primitive part: the parasympathetic nervous system. The brain processes information in the hypothalamus region, and the ANS sends signals to the rest of the body to either stimulate or relax different functions. Auto-responses from the ANS are elicited in the event of stress, fragmented sleep, unhealthy diets and other chemical or neural factors affecting a person's resting state. HRV is a non-invasive way to identify ANS imbalances, as when the nervous system is behaving unusually, the variation in the heartbeats is relatively more erratic. A higher HRV score generally indicates better cardiovascular fitness and resilience to stress. In comparison, a lower HRV score is associated with an increased risk of cardiovascular health and mental health concerns [9].

The primary feature used in HRV calculations is the time between each successive heartbeats, or the time between successive normal or abnormal QRS complexes/peaks in milliseconds, defined as the R-R peak interval. Estimation of the R-R interval involves first detecting the QRS complexes/peaks and subtracting the observed times of successive peaks. It should be noted that a distinction is made between R-R intervals, and the typically synonymous N-N interval, as the latter only accounts for normal-normal beats, while the former accounts for normal-normal, normal-abnormal, or abnormal-abnormal cases.

PRV is used to measure the similar inter-beat variation property with PPG signals, and this denotes the pulse-to-pulse variation in time. PRV quantifies approximately the same behavior as the intervals between successive R peaks or QRS complex observed in ECG with the systolic peak-to-systolic peak or diastolic peak-to-diastolic peak intervals.

Malik et al. [28] observed the potential of HRV in assessing the role of ANS fluctuations in normal healthy individuals and those with diseases. Relevant measures were selected from the previous research and used as HRV features for the scope of this work.

This work primarily uses the formulas shown by Equations (1) and (2) to calculate Root Mean Square of Successive Differences between the R-R intervals (rMSSD) [28] and Standard Deviation of RR intervals (SDRR) [28]:

$$rMSSD = \sqrt{\frac{\sum\_{i=1}^{N-1} \left( RR\_i - RR\_{i+1} \right)^2}{N-1}} \tag{1}$$

From Equation (1), N is the number of R-R intervals and *RRi* is the location of the *i*th QRS complex/peak observed at a time in milliseconds.

$$SDRR = \sqrt{\frac{1}{N-1} \* \sum\_{j=1}^{N} \left( RR\_j - \overline{\mathcal{R}\mathcal{R}} \right)^2} \tag{2}$$

From Equation (2), N is the number of R-R intervals, *RRj* is the location of the *j*th QRS complex/peak observed at a time in milliseconds.

The features of rMSSD and SDRR respectively reflect the number of fluctuations in heart rhythms and the degree of variation between heart beats. Hence, both are vital features to consider when aiming to predict the cardiovascular state. Various cardiac conditions were detected using short-term HRV features, with rMSSD, SDRR, and pRR50 being the most useful in predicting changes in parasympathetic activity and even being a possible indicator of cardiac mortality [29]. Additional HRV features are also included in Table 1 and used in this work. Those additional features include the coefficient of variation in R-R intervals (CVRR) and coefficient of variation in the differences of successive R-R intervals (CVSD), as they are features that improve the classification of CVD [30]. Researchers recorded PPG signals from the fingertips of subjects extracted PRV features, such as rMSSD, SDRR, and pRR50, compared them with the same features obtained from ECG to validate the accuracy, and found that the average error rate was less than 6% [30]. Another study used wearables to compare the time domain features (rMSSD, SDRR) of HRV extracted from ECG and PRV extracted from PPG signals and found that PPG signals can be used as an alternative source for HRV measurement [31]. The features used in this work are presented in Table 1.

**Table 1.** A summary of the HRV feature characteristics used in this work.

This relationship can be utilized to monitor individuals' cardiovascular health with off-the-shelf sensors for classifications and early detection of diseases. The commonality between the behavior of the HRV and PRV parameters can be utilized to enable generalized detection of AF across two different modalities: ECG and PPG. The model is trained on HRV features derived from ECG signals within the three ambulatory datasets. The model is tested and finetuned on PRV features derived from PPG signals within the wearable dataset.

For verification of the created dataset and its respective HRV values for different R-R interval measures, it was compared to the short-term normative values reported in [32], and the reference ranges for HRV from ECG recordings [33]. The HRV features of NN intervals, rMSSD, and SDRR were the most reported along with their normative ranges, and it is shown in Table 2. This comparison ensured that the extracted PRV features for real-time samples from the low-cost PPG sensors from wearables were within reasonable bounds of the ground truth cases and should remain valid for this experimentation.

**Table 2.** HRV Reference Ranges.

#### **3. Proposed Approach**

The proposed approach has three main stages after the initial acquisition of datasets, as shown in Figure 3. The first stage involves preprocessing the signals in terms of filtering, peak detection, and feature extraction. The second involves the one-dimensional convolutional neural network (CNN) model development for binary classification between NSR and AF with temporal HRV features, and trained with the ECG datasets. The third stage involves model evaluation. The model evaluation is done on both the holdout testing on a subset from the ECG datasets and out-of-sample cross-domain testing instances from the PPG datasets. Each stage is detailed in the following subsections.

**Figure 3.** High-level view of the proposed approach.

#### *3.1. ECG Datasets*

NSR and AF rhythms are collected from three datasets [34]: MIT-BIH Normal Sinus Rhythm (NSR-DB), MIT-BIH Atrial Fibrillation (AF-DB), and MIT-BIH Arrhythmia (ARR-DB).

Using ambulatory ECG recorders, each record was acquired from patients referred to the Arrhythmia Laboratory at the Beth Israel Deaconess Medical Center, Massachusetts Institute of Technology. They are accessible via the Physiobank repository, a digital archive of well-characterized biomedical signals created by the United States National Institutes of Health for use by the research community [35].

AF-DB consists of 23 two-channel ECG recordings (sampled at 250 Hz), from subjects with paroxysmal atrial fibrillation, atrial flutter, AV junctional rhythm, and normal rhythms, with a typical recording bandwidth of approximately 0.1 to 40 Hz. NSR-DB consists of 18 two-channel ECG recordings (sampled at 128 Hz) from subjects with no significant arrhythmia or heart abnormalities. ARR-DB consists of 48 records, each containing twochannel ambulatory ECG signals of 30-min duration. Lead 1 channel ECG signals, which record the right ventricle and right atrium, are used in this work.

The signals in AF-DB have rhythm annotations indicating NSR and AF. Meanwhile, the signals in NSR-DB and ARR-DB have heartbeat annotations as well, in addition to rhythm annotations for AF and NSR. The annotations are provided in terms of a distinct beginning and end label pertaining to particular regions of the signals. The heartbeats in NSR-DB and ARR-DB follow the recommended standards of the Association for the Advancement of Medical Instrumentation [36]. Hence, the annotations/labels for each heartbeat in the NSR-DB and ARR-DB fall into multiple categories [37]. The beat superclasses and their corresponding beat annotations of interest in this work are N: (*N*, *L*, *R*, *B*) and S: *A*, *a*, *J*, *S*, *j*, *e*, *n*. While the primary focus is on heart rhythm classification, specific samples in the dataset are considered on a heartbeat segment basis for incorporating cases of atrial premature complexes (APC) [38]. The rationale for incorporating heart rhythms with high saturation levels of anomalous heartbeats is to contribute stochasticity (diversity) to the AF class. The expectation is that the dataset consisting of contiguous AF rhythms and AF rhythms interspersed with normal and other types of beats will allow for the eventual detection of varying anomalous rhythms that differ considerably from the purely NSR training samples [39,40].

#### *3.2. PPG Dataset*

The privately held UMass PPG database (UMass-DB) [41] collected by the University of Massachusetts Medical School was used for further testing to discover the strengths and weaknesses of the model. The authors of [42] granted access to this dataset and consists of 37 subjects, with 10 having AF. The PPG signals were recorded at a sampling frequency of 128 Hz from the Simband, smart wristwatch provided by Samsung, which has 8 PPG sensors, a triaxial accelerometer, an ECG lead, and a temperature sensor [42].

Figure 4 presents the typical characteristic heart rate rhythm samples from both datasets reflecting NSR and AF, respectively, across the ECG and PPG modalities. As observed from Figure 4a,c,e, NSR instance is a normal heart rhythm that maintains a steady rate with no irregularities. From Figure 4b,d,f, the AF instance is a sustained unsteady heart rhythm with rapid fluctuations.

**Figure 4.** Sample 30-s heart rhythm instances represented as raw amplitude (y-axis) against time (x-axis) from ECG datasets; (**a**) NSR ECG from AF-DB (Patient 4015); (**b**) AF ECG from AF-DB (Patient 4043); (**c**) NSR ECG from ARR-DB (Patient 100); (**d**) AF ECG from ARR-DB (Patient 222) and PPG dataset; (**e**) NSR PPG (Patient 4002); (**f**) AF PPG (Patient 4012).

#### *3.3. Preprocessing*

Initially, the signals with rhythm annotations of NSR and AF from AF-DB, ARR-DB, and NSR-DB were divided into 30-s samples with no-overlapping windows. The segmented 30-s signals retained the respective label of NSR or AF as multiple 30-s samples can be obtained from a single longer signal with the same annotation. In the case of ARR-DB, all signals with annotations corresponding to non-atrial complications, such as paced rhythms, ventricular bigeminy, trigeminy, tachycardia, were ignored.

Most AF contiguous data samples originated from the AF-DB, with approximately 3.6% being from the ARR-DB dataset. From the NSR database, 15% of the total NSR rhythm records were arbitrarily selected. Most NSR data originated from NSR-DB, followed by ARR-DB while AF-DB contributed only 5% of the total NSR samples. All the signals accounted for had the highest resolution in terms of QRS complex certainty.

In addition, signals with ARR-DB were examined further in terms of heartbeat saturation to determine the presence of excessive supraventricular activity, which is associated with an increased risk of developing atrial fibrillation [43]. The examined signals were annotated with APC, supraventricular tachyarrhythmia (SVTA), atrial couplets, or atrial

flutter. As per AAMI standards, all considered heartbeats in the 30-s window derived from these signals belonged to the class N or S. The beats denoted by S can be referred to as supraventricular ectopic beats or premature beats. Although ectopic beats are mostly harmless, recent studies have shown that frequent repetitions of supraventricular ectopic behavior can indicate the presence of potential atrial abnormalities [44].

The criteria for judging the label of a 30-s rhythm are based on the saturation level of class S beats. If zero S beats are present, then it is ignored, and if over 50% of the beats are S with an annotation of a, *J*, *A*, *S*, *j*, *e*, or *n*, it is treated as an AF rhythm. The passage from heartbeat types to heart rhythms is not necessarily direct. Thus, this rule is to ensure that only segments consisting of non-isolated beats are treated as AF samples.

Individuals in real scenarios may not always exhibit signs of sustained arrhythmia. It is possible for a fluctuating pattern between normal rhythms, where relatively shorter (<30 s) intermittent periods of abnormal heart behavior associated with AF can be observed, and thereby contributing to AF risk stratification. Excessive ectopic activity can cause palpitations, light-headedness, and increased awareness of heartbeats [45]. For instance, patient 232 does not have any AF rhythm annotations, but has frequent ectopic runs. The cardiologists' notes associated with the annotated record of patient 232 report the presence of sick sinus syndrome, which is an abnormality in the right atrium of the heart. To address this case of potential variability in patients and boost the robustness in classification performance of the developed model, instances that are not solely NSR but anomalous to a considerable degree were treated as an AF class instance.

As per the findings of [27,46], a second-order Butterworth filter was applied with the bandpass frequencies of 8Hz–20Hz for removing baseline drift, motion artifacts and minimizing other ECG features such as the P and T waves. The signals of the MIT-BIH Arrhythmia, MIT-BIH NSR, and MIT-BIH AF databases have sampling rates of 360 Hz, 128 Hz, 250 Hz, respectively. Fast Fourier (FFT) resampling is applied to down-sample the signals to 50 Hz, as the signals from the three MIT-BIH databases have different original sampling rates. It, therefore, must have the same frequency before any further processing. The method reported in [46] achieves the highest signal-to-noise ratio and optimal QRS complex detection on the MIT-BIH databases instead of techniques such as the Pan Tompkins algorithm [47], and the former method is utilized to produce a list of the peaks necessary to derive the time-domain HRV features.

PPG signal filtering was conducted with a 3rd order Butterworth filter with 0.5 Hz and 8 Hz cutoffs to remove powerline interference, motion artifacts, and other saturated noise [48]. The UMass dataset signals were down-sampled from 128 Hz to 50 Hz using FFT resampling, similar to the approach executed in [42]. Systolic peak detection in the PPG signals utilized the algorithm outlined in [49], where two event-related moving averages with an offset threshold empirically yielded higher accuracy than the alternative techniques of Billauer [50], Li [48], and Zong [51].

The decision for down-sampling all signals to 50 Hz, instead of up-sampling any acquired signals to 128 Hz is based on two key factors. Firstly, most PPG based devices do not have a high sampling rate (~128 Hz), and vary from 60 Hz to 100 Hz based on the quality of the sensor and the battery levels of the device the sensor is embedded in. However, the minimum sampling frequency required is 50 Hz to derive reasonably accurate HRV and PRV parameters with a low margin of error from ECG and PPG signals, respectively [52,53]. Secondly, the computational overhead is reduced without a significant effect on the signal acquisition or processing aspects, which can extend the deployment of the proposed model in this work to resource-constrained wearable devices.

It is to be noted that the systolic peak detection algorithm for PPG signals proposed in [48] is a modified variant of the QRS peak detection algorithm for MIT-BIH database ECG signals proposed in [46]. This work performed filtering as per the recommended cutoff frequencies before applying the algorithm, as mentioned previously in this section. The general description of the algorithm reported in [46,48] is as follows:

After performing the peak detection algorithm summarized in Algorithm 1, a list of peak locations and their occurrence times enables the estimation of RR intervals or systolic peak-to-systolic peak intervals. From the intervals, the time-domain HRV and PRV features are derived in terms of their statistical characteristics as described in Table 1.

**Algorithm 1.** Pseudocode of peak detection algorithm and feature detection for dataset *D*.

```
FOR xi in D (ECG or PPG data instance from dataset, where i = {0 . . . size(D))
Filtered signal S[n] = BandpassFilter(xi)
Let peaklist = {} (Peak amplitudes)
Let timelist = {} (Peak times)
Let BlocksO f Interest = {}
Let yi = {}
Set W1 = Average ECG or PPG peak duration
Set W2 = Average ECG or PPG beat duration
Set MApeak = MovingAverage(S[n], W1)
Set MAbeat = MovingAverage(S[n], W2)
Set threshold α = W1 + β
FOR n = 1 to length(MApeak)
IF MApeak[n] > MAbeat[n] THEN
                              BlocksO f Interest[n] = 1
ELSE
                              BlocksO f Interest[n] = 0
END IF
END FOR
```

```
FOR j = 0 to length(BlocksOfInterest)
IF width(BlocksO f Interest[j]) ≥ α THEN
peaklist[j] = max(BlockOfInterest[j])
timelist[j] = time(BlockOfInterest[j])
ELSE;
                                   reject block
END IF
END FOR
{rMSSD, ..., pRR50} = Calculate HRV/PRV (peaklist, timelist)
Transformed data instance yi = {rMSSD, ..., pRR50 }
Save yi to updated dataset D
END FOR
```
Finally, Z-score normalization is performed on the derived features. All ECG and PPG datasets signal instances are fixed with zero mean (μ = 0), and unit standard deviation (σ = 1.) This step mitigates amplitude scaling issues, offset effects, and reduces drastic variability in the signal values. Table 3 presents total data samples of NSR and AF classes after pre-processing.

**Table 3.** Total data Samples of NSR and AF classes after preprocessing.

#### *3.4. Model Development*

The model developed in this work is a one-dimensional 12-layer CNN for the classification of NSR and AF. The proposed architecture for the CNN is depicted in Figure 5, outlining the input tier, model tier, and output tier. The model receives temporal HRV features extracted from ECG signals as input, propagates them through the neural network, and outputs a single output indicating whether the input instance belongs to NSR or AF class. A detailed summary of the CNN properties and parameters is listed in Table 4. The configuration of the layers and their respective parameters reported were attained after hyperparameter tuning through GridSearch.

A single model is selected after training and evaluation. It is trained and tested using the HRV features derived from ECG, and finetuned to classify AF with PRV features derived from PPG. Due to the inherent similarities between the statistical properties of HRV and PRV, this approximation makes it possible for a unified AF representation across two wearable modalities.

There are three types of layers within a CNN: convolutional, pooling, and fully connected layers. An instantiated convolutional layer detects local conjunctions of features from a preceding layer which can be either an input layer or another convolutional layer. The convolutional layer merges semantically similar input features into a single learned representation. It is to be noted that features in the context of the neural network imply semantic similarities or overarching patterns detected across the provided inputs (a unified vector of HRV features). Receptive fields in each convolutional layer focus on different aspects of the derived features to create their internal representation of the inputs. The property of shared weights ensures that general features common to all data samples are learned once and shared with the other convolutional layers in the network. Subsampling reduces the dimensionality of the data to identify the most significant features. This can be related to size (spatial) or time sequence (temporal). A set of weighted vectors known

as a filter/kernel outputs feature maps based on local receptive fields at each layer. These feature maps usually hold general characteristic information inferred from input feature data samples at a particular layer by the neural network [54].

**Figure 5.** Architecture for the proposed CNN model.

**Table 4.** Summary of properties for the proposed CNN model.

Each layer of the proposed CNN architecture and the components of activation and regularization presented in Figure 5 are described as follows:

1. Convolutional Layer (Conv1D): In this layer, a convolution operation using Equation (3) is performed by sliding the filter/kernel over the input features to obtain a feature map as the output.

$$\mathcal{L}\_m = \sum\_{n=0}^{N-1} f\_n k\_{m-n} \tag{3}$$

From Equation (3), *k*, *c*, *f*, and *N* denote the inputs, filter/kernel, the output feature map, and the number of elements in input *k*, respectively. In the CNN model developed for this work, there are four convolutional layers with 256, 128, 64 and 32 filters, respectively. The filter dimensions used in this layer are 5 × 5, which yielded the best result.

2. Fully Connected Layer (FC): This layer compiles the results obtained from the preceding convolution and pooling layers to estimate an output classification label using Equation (4) [55]:

$$\mathbf{x}\_{i} = \sum\_{j} w\_{ji} y\_{j} + b\_{i} \tag{4}$$

From Equation (4), *w* and *b* denote weights and biases, respectively. Here, *y* is the output from a previous layer *j* and *x* is the output of the current layer *i*. In the CNN model developed in this work, there are two fully connected layers, with 8 and 1 neurons, respectively.

	- Rectified Linear Unit (ReLU) [56]: This is the activation function that is used in all three convolutional layers of the network. The Rectified Linear Unit produces 0, as an output *x* < 0, and then produces a linear output with slope 1, when *x* > 0. It introduces non-linearity and mitigates the vanishing gradient problem, which is where the lower layers of the network train slowly as the gradient of optimization decreases exponentially. This leads to sparse neuron activation, more straightforward output, and makes computations easier while preserving the significant receptive fields of the convolution layers.
	- Sigmoid [57]: An activation function used in the second fully connected layer, with 1 neuron. Sigmoid activation functions are monotonic and differentiable. Their mathematical property maps real number values to the [0, 1] range to render the output as a probability, given the particular set of transformed input HRV features. In this work, the binary classification output of 0 indicates that an instance belongs to the NSR class, and 1 means that it belongs to the AF class.
	- Batch Normalization (BN) [59]: This technique reduces the covariance shift, meaning that minor features differences that do not contribute heavily to the overall model performance will not be considered with high priority. Therefore, minor changes between the ranges of training data, validation data, or unseen data will not affect the classification performance and allow each layer to be more independent about certain input features.

• Dropout (DP) [60]: This technique randomly drops neurons and their connections to prevent neurons from co-adapting. This makes each neuron more responsible for capturing the overall data representation and contributing to the final output. The dropout rate, which reflects the percentage of random neurons to be dropped, was set to 0.2.

#### *3.5. Training and Testing*

The CNN model is trained with the back-propagation algorithm [54] with a minibatch of 16. According to [61], taking a subset of the entire data for each epoch improved generalization performance and had a smaller memory footprint. An epoch is the number of times the training set passes through a neural network completing a feed-forward and back-propagation phase. In this work, the total number of epochs was 50. The Adaptive Moment Estimation (ADAM) [62] optimizer was used for effective training convergence.

From the dataset, 80% was randomly divided for training and validation, and 20% was used as the test set. The Stratified k-fold cross-validation strategy was implemented with k = 5 [63]. In each fold, the training and validation subset is randomly divided into 5 equal parts, where with cross-validation, each data instance is used for both training and validation. Stratified k-fold cross-validation ensures that the class distribution in each of the five equal parts remains consistent across iterations to address potential biases. This was conducted to observe the generalizability and variability of the developed model to reflect its performance with new data. The 20% testing subset serves as the holdout data that the model has not been trained/validated with.

#### **4. Results**

This section describes the environment setting, reports the achieved diagnostic performance measures of the proposed convolutional model neural network on the ECG training data and unseen PPG data. To assess the implementation feasibility of the developed model, it was interfaced with a smartphone application and integrated within a health monitoring context.

#### *4.1. Implementation Environment*

The proposed CNN algorithm was implemented on a workstation with Windows OS, an Intel Kabylake 2.80GHz processor (i7-7700HQ), and 16 GB of RAM. The time required for training and testing the CNN model with 50 epochs was approximately 4420.67 s. The deep learning platform employed in this work was Keras [64], a high-level neural networks framework with a Tensorflow backend [65]. The Waveform-Database Package (WFDB) published by Physionet was used to directly access the MIT-BIH Arrhythmia dataset [35], consisting of heart rhythm samples and their respective annotations. The Sklearn module was used for data preprocessing and normalization operations [66]. Neurokit (NK), a toolbox for statistics and neurophysiological signal processing, was used to extract the ECG and PPG time-series features [67].

#### *4.2. Model Evaluation on ECG Datasets*

The diagnostic performance measures of accuracy, sensitivity, specificity, F1-score, and AUC are evaluated on a holdout test set in each of the five folds. Accuracy is the proportion of true outputs with respect to all data instances. Sensitivity is the model's ability to classify data instances belonging to a certain class correctly. Specificity is the model's ability to correctly distinguish data instances that do not belong to specific classes. F1-score is the harmonic mean between precision (ratio of correctly distinguished positives over all predicted positive) and recall (sensitivity), and the area under the curve (AUC) measures the quality of binary classification outputs in terms of sensitivity against false positive rate. To develop high-fidelity biomedical models as the proposed approach, high sensitivity and specificity are vital. They gauge the model's ability to correctly detect

patients with a certain cardiac arrhythmia and correctly detect patients without cardiac arrythmia [68].

To calculate the measures as in Equation (5), the model classification outputs must be quantified in terms of True Positives (TP), False Positives (FP), False Negatives (FN) and True Negatives (TN) [69].

$$\begin{array}{l} Accuracy = \frac{TP\_{NSR} + TN\_{NSR}}{TP\_{NSR} + TN\_{NSR} + FP\_{NSR} + FN\_{NSR}}\\ Sensitivity = \frac{TP\_{NSR}}{TP\_{NSR} + FN\_{NSR}}\\ Specificity = \frac{TN\_{NSR}}{TN\_{NSR} + FP\_{NSR}}\\ F1Score = \frac{TP\_{NSR}}{TP\_{NSR} + (0.5\*(FP\_{NSR} + FN\_{NSR}))} \end{array} \tag{5}$$

Let *Yi j* be the data instances where *i* is the true class, *j* is the predicted class, and *i*, *j* ∈ {*NSR*, *AF*}. Consider the class AF signifying atrial fibrillation rhythms, and then, its outputs are defined as follows:

The aggregated scores across all 5 folds are summarized in Table 5, and exhibit a high AF classification performance. The true positive (*TPAF*) rate is 96.90%, and the true negative *TNAF* rate is 95.13%.

**Table 5.** Aggregated classification metrics across five-folds expressed as mean and standard deviation.

#### *4.3. Model Evaluation on PPG Dataset*

While evaluating the model on the PPG dataset, two scenarios are considered. In the first scenario, the weights of the pre-trained model were not updated through transfer learning. In the second scenario, the model was finetuned by retraining the PPG signals.

In the first scenario, the model correctly classified 170 out of 192 samples of NSR, and 42 out of 54 samples as AF. The true positive (*TPAF*) rate is 77.80%, and the true negative (*TNAF*) rate is 88.54%. The measures reported in Table 6 serves as an initial benchmark test to gauge the performance of the ECG HRV trained on PPG data that have not been encountered during training or validation by the CNN model.

**Table 6.** Performance measures of the ECG-trained model on the complete UMass-DB PPG signals before transfer learning.

In the second scenario, the learned weights of the model are updated by using 75% of UMass-DB for (60%) training and (15%) validation, with 25% for holdout testing, following the Stratified k-fold cross-validation with k = 4. By employing this approach, the intention is to adapt the weights of the pre-trained CNN model with 75% of the PPG data instances, test its performance on the remaining 25% of the untrained PPG data instances. This was applied four separate times, such that every instance is used for training, validation, and testing independently without data leakage between the training/validation and the testing sets. The aggregated testing performance is reported in Table 7, where the model makes predictions on all instances fairly.

**Table 7.** Performance measures of the ECG-trained model on UMass-DB PPG signals after transfer learning folds expressed as mean and standard deviation.

After retraining, the average true positive (*TPAF*) rate is 94.33% and the average true negative *TNAF* rate is 95.20%.

This performance is considerably high as the model classifies instances from a different input modality (PPG), when it was trained using only ECG signals. A marginal increase in performance is observed when transfer learning is implemented. Type I and type II errors were also observed, at a lower degree, resulting in AF false positives and AF false negatives, as shown by the results in Tables 6 and 7. This indicates that the boundaries between the NSR and AF to a certain extent are not clearly distinct in both the ECG and PPG recordings. Factors, such as PPG sensor specifications, reliability, and quality, may contribute to the decreased classification measures compared to the training performance. It is to be noted that both the ECG training samples and the PPG samples were resampled using FFT at 50 Hz, 100 Hz, 128 Hz, 250 Hz and 360 Hz, corresponding to the different sampling rates of the original dataset recordings to see the differences in the achieved results. The conducted empirical experiments found that 50 Hz for all recordings yielded relatively similar performance when classifying PPG signals as ~128 Hz (the minimum sampling rate across all datasets).

#### *4.4. Implementation and Testing*

In addition to the validation conducted in Section 4.3, a prototype implementation was further developed and tested on live human subjects. The developed model was integrated within a health monitoring platform to test and ascertain its real-world performance. A smartphone application was designed to acquire PPG recordings, interface with the model, and retrieve predictions of AF from human subjects.

The system that implements the proposed CNN model presented in this work was realized by following the three-tier architecture for modularity, scalability, and testing. The model was deployed via a Python Flask [70] server with a Google Firestore [71] database on the same workstation. Figure 6 presents the smartphone application collecting the input from the sensor, i.e., raw PPG heart rhythm values and sending an HTTP POST request to the REST API server containing the recorded heart rhythm values. The smartphone application receives a response from the server (end-to-end response time ≈ 1.25 s) indicating whether the recording was NSR or AF.

**Figure 6.** Implementation Architecture.

The PPG signals from most variations of optical sensors available in general-purpose smartphones and wearable devices can be used in the classification of AF after applying the techniques of filtering, down-sampling, peak detection, PRV extraction as outlined in Section 3.3. The specifications of the particular sensor used in this implementation are listed in Table 8 and have a maximum frequency of 100.0 Hz. The sensor type is 65,572 and is manufactured by MAXIM. The heart rate monitor LED measures the magnitude of the red light reflected from an individual's blood vessels at the measurement location, in the range of 0–350,000 (unitless). It operates on a 3.0 V to 5.5 V single supply voltage, with dimensions of 2.9 mm × 4.3 mm × 1.4 mm, and is integrated into portable or wearable devices. The devices used in the experiments were the Samsung S9, Samsung Note 8, and Samsung Note 9. M. Elgendi et al. [72] used Samsung 9th generation smartphones, the same ones used in this work.

**Table 8.** Sensor specifications for the smartphones used in this experiment.

The prototype implementation was successfully verified on the human subjects with the complete flow from signal acquisition to live AF classification following the same preprocessing techniques for filtering and resampling used for the UMass-DB PPG signals.

The human subjects were classified into healthy human subjects with no reported medical conditions, while the other was a heart patient from the Welcare Hospital Ernakulam, India. To record the heart rhythm, the subject is required to position their fingertip on the smartphone's heart rate sensor. Upon the detection of the PPG input signal, the smartphone application initiates the PPG value acquisition process. The healthy subject continues to hold their fingertip in place for 30 s, and then, the signal is transmitted to the server. The model classified one of the short length heart rhythms obtained at rest as NSR, as shown in Figure 7a. The heart patients' vitals are supervised through a bedside monitor by the doctor. Upon detecting an oncoming abnormality on the monitor, the patient is asked to place their finger on the smartphone and record a PPG signal. The result is shown in Figure 7b. The classification is saved in the cloud database under a specific entry for each subject, and the REST API server processes and responds to each acquired signal. This allows subjects and doctors to access historical records of the subject heart activity regularly.

**Figure 7.** Cardiac arrythmia classifications presented on a smartphone application by the model using live PPG readings from the fingertip smartphone sensor; (**a**) presents NSR Classification; (**b**) presents AF Classification.

The healthy subject underwent a Treadmill Stress Test in the clinical laboratory to observe the similarity in heartbeats and peak formations between an ECG and the PPG peak detection algorithm used in this work. The Treadmill Stress Test uses medical-grade multi-lead ECG to capture heart activity to measure cardiovascular health. A reference ECG signal was simultaneously collected to validate the PPG signal obtained from the smartphone sensors for the same 30 s. The resulting waveforms are shown in Figure 8a,b. Both Figure 8a,b estimate the same BPM, indicating potential consistency in the number of detected peaks.

**Figure 8.** Measured ECG vs. estimated PPG BPM comparison. (**a**) Reference ECG-measured BPM; (**b**) PPG-derived BPM by peak detection algorithm.

#### **5. Discussion**

This study explored the efficiency of using convolutional neural networks to classify short-length heart rhythms using the concept of HRV-derived features to generalize AF representation across both the ECG and PPG modality. In this paper, the proposed model is compared and contrasted with similar works in the literature. The primary contributions of this research are highlighted in the following subsections.

#### *5.1. Comparison with Existing Works*

Table 9 presents recent advances in the literature for short-length cardiac arrhythmia detection using one or more HRV features with applicability in portable devices.

Zhou et al. [17] employed a modified version of the Shannon entropy algorithm for AF detection by constructing symbolic sequences and probability distributions using ECG-based R-R intervals from the MIT AF database. This statistical approach was one of the first studies to discuss the possibility of deploying such approaches in portable devices. Islam et al. [73] presented a rhythm-based heartbeat normalization technique for improved ECG-based AF detection by measuring irregularities in a specified window of heartbeats. The datasets used for training and testing were the MIT-BIH AF database and MIT-BIH Arrhythmia, respectively. Cui et al. [18] proposed a similarity analysis and ensemble scheme that maps R-R intervals to binary symbolic sequences and compares the rank-frequencies to quantify the differences between AF and NSR using the ECGbased MIT-BIH AF database. Shashikumar et al. [74] presented one of the first and few works proposing cross-domain generalizability of cardiac arrhythmia models and used Bidirectional Recurrent Neural Network for AF detection from a single lead ECG. The researchers collected the ECG dataset from the University of Virginia Heart Station, United States, for training and collected the PPG dataset from the Emory Hospital and Grady Memorial Hospital, Atlanta, United States, for testing. They reported high classification performance for the cross-domain application using spectral features and R-R time series features with wavelet decomposition. Bashar et al. [75] utilized support vector machines

on 30-s-long PPG signals for AF and NSR detection. They trained and tested on a custommade PPG dataset and addressed noise saturation by using Butterworth filters. Tarniceriu et al. [76] implemented Markov models to detect AF and NSR by using R-R intervals as features and collected a dataset with a custom wearable prototype. Aliamiri et al. [77] employed an end-end deep learning PPG-based AF detection system that filters poor quality signals. They developed a convolution-recurrent hybrid model using waveform features on a custom-made PPG dataset that could effectively distinguish between AF and NSR. Tison et al. [78] conducted one of the first large-scale studies for passive AF detection using PPG-enabled smartwatches in collaboration with the University of California, San Francisco and Cardiogram. Cardiogram is an Apple watch application used to obtain heart rate data. The researchers used these collected data to implement a deep neural network with heuristic pretraining and R-R intervals as a feature set. Fallet et al. [79] utilized decision trees with waveform features and RR-intervals to classify AF and ventricular arrhythmia in 10-s-long PPG signals. The researchers created a PPG signal dataset from Lausanne University Hospital Switzerland and used a custom wearable prototype to test their results. Kwon et al. [80] employed a 1D CNN to process 30-s-long PPG signals to classify AF and NSR with a custom-made dataset.

**Table 9.** A comparison of recent works developed for CVD detection with machine learning and portable devices.

The performance measures obtained in this work are competitive with the works reported previously. The existing research has achieved successful results in the domain, however, has a few limitations that the proposed approach in this paper addresses. Firstly, the PPG datasets are not gold-standard and are not publicly accessible to reproducible and further testing. In this work, the reputed MIT-BIH datasets are utilized for implementing a cross-domain generalizable model. The input features of HRV captures a holistic representation of cardiac activity, as they are the most consistent medium of commonality between ECG and the PRV aspect of PPG signals. Secondly, existing models trained on ECG signals cannot be applied to predict PPG directly due to the differences in their morphology. In most of the works, ECG-based models can only work with portable devices having ECG sensors, and the PPG based-models require custom wearable prototypes or hospital settings, except in [78]. Thirdly, the developed models are not trained with multiple datasets or assessed on unseen data, lowering the likelihood of being applicable

in non-ambulatory settings. Lastly, this work provides a supplementary approach, wherein the time-domain HRV representations are extracted from larger public datasets instead of raw signals, which extends the applicability to both ECG and PPG derived from clinical devices or consumer wearables.

#### *5.2. Research Impact*

This work presents a generalizable approach that has the potential for sensor agnostic CVD classification. The model is trained on data acquired from the source ECG modality and finetuned by updating the learned parameters using data from the target PPG modality. There were 15,434 instances from the ECG datasets of both NSR and AF for training the model, while there were only 192 total instances from the PPG dataset. Through the development of models with large cohorts of data in the related domain of ECG and the use of transfer learning, the issue of limited, gold-standard data accessibility from consumer wearable devices can be resolved. This can enable healthcare providers to leverage such devices in conjunction with cardiac arrhythmia classification models for non-ambulatory cardiovascular prognosis in the general population.

Smart healthcare platforms are holistic systems that enable disease prevention, monitoring, diagnosis, and treatment and connect patients with medical professionals. These are significant risk factors for the progression of CVD in patients. Repeated detection of any cardiovascular impairments as indicated by the AF in this work can prompt a clinical checkup, thereby allowing for early treatment and outcome improvement. A systematic survey by Majumder et al. [81] of 11 smartphone cardiac monitoring applications showed that the majority of them used simple, static heart rate threshold-based risk stratification. Furthermore, the existing solutions were not designed to be part of a monitoring system that can interface with clinicians but rather limited to the device only within the scope of the testing setting. Kakria et al. [82] proposed a real-time cardiac health monitoring system with a patient and doctor portal for effective monitoring using a custom Bluetooth wearable device and smartphone. However, medical alerts sent to patients and users lacked specificity, as any heartbeat above or below a threshold is flagged as abnormal. Moreover, there were no considerations for noise saturation or adaptability to signals other than PPG. In resource-constrained settings such as inexpensive fitness bands, extracting only the features necessary instead of complete raw signal samples can prove to be more efficient, as demonstrated in this work.

A possible limitation stems from the fact that there appears to be an overlap between the samples of each class. This could be due to the differences in resting heart rates across individuals, general fitness levels, and the influence of underlying health conditions. A direct approach to boost the model's performance is to incorporate additional real ECG samples from more reputable datasets. Finally, spectral and non-linear HRV measures [83] can be added to the feature space to capture more robust representations of each class.

#### **6. Conclusions**

This work proposed a design and implementation of an explainable deep learning 1D-CNN model for use in smart healthcare systems with general-purpose devices such as smart wearables and smartphones. The 1D-CNN model classifies the NSR and AF from short length ECG or PPG signals using HRV features as inputs with the MIT-BIH ECG datasets.

The 1D-CNN model achieved overall classification performances with accuracy of 95.50%, sensitivity: 94.50%, specificity: 96.00%, F1-score: 93.40%, and AUC: 95.30% across a five-fold cross-validation approach. In comparison to other works in the literature, these performance measures are highly competitive and can be integrated into mobile health monitoring platforms with general-purpose devices. Thereby, the proposed approach is one of the first works to develop a cross-domain generalizable ECG-based model for deployment in smartphones and wearable devices.

Furthermore, the proposed methodology removes noise and motion artifacts from commercial PPG-sensors within a framework for health monitoring, thereby making early detection systems accessible for the general public. This approach brings to the forefront the applicability of ECG databases to enable machine learning to transform the PPG sensor readings from commercial devices. This can mitigate the issues of developing classification models that can only be used in controlled settings as well as increase the types of cardiac arrhythmia that can be observed from general-purpose devices and eliminate difficulties associated with creating custom PPG datasets for each study.

Subsequent research directions involve conducting a longitudinal study for exhaustive testing with users to attain additional empirical evidence supporting the real-world applicability of this approach, benchmarking the model against further gold-standard datasets, and extending the scope of the health monitoring framework.

**Author Contributions:** Conceptualization, J.R., R.A. and A.S.; data curation, J.R. and Z.S.; investigation, J.R. and Z.S.; methodology, J.R., R.A. and A.S.; software, J.R., Z.S. and R.A.; supervision, R.A. and A.S.; writing, J.R., Z.S., R.A. and A.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was partially funded by the American University of Sharjah faculty research grant FRG17-R-20. The work in this paper was supported, in part, by the Open Access Program from the American University of Sharjah. This paper represents the opinions of the author(s) and does not mean to represent the position or opinions of the American University of Sharjah.

**Institutional Review Board Statement:** No IRB required.

**Informed Consent Statement:** A consent has been obtained from the patient(s) to use this dataset for research purposes and publishing this paper.

**Data Availability Statement:** The ECG dataset adopted in this research is openly available in [Physionet] at https://doi.org/10.13026/C2F305, https://doi.org/10.13026/C2MW2D, and https: //doi.org/10.13026/C2NK5R (accessed on 1 September 2021). The PPG dataset analyzed in this research is openly available in https://www.synapse.org/#!Synapse:syn23565056/wiki/608635 (accessed on 1 September 2021).

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

### *Article* **Non-Contact Spirometry Using a Mobile Thermal Camera and AI Regression**

**Luay Fraiwan 1,2,\*, Natheer Khasawneh 3, Khaldon Lweesy 2, Mennatalla Elbalki 1, Amna Almarzooqi 1 and Nada Abu Hamra 1**

**Abstract:** Non-contact physiological measurements have been under investigation for many years, and among these measurements is non-contact spirometry, which could provide acute and chronic pulmonary disease monitoring and diagnosis. This work presents a feasibility study for non-contact spirometry measurements using a mobile thermal imaging system. Thermal images were acquired from 19 subjects for measuring the respiration rate and the volume of inhaled and exhaled air. A mobile application was built to measure the respiration rate and export the respiration signal to a personal computer. The mobile application acquired thermal video images at a rate of nine frames/second and the OpenCV library was used for localization of the area of interest (nose and mouth). Artificial intelligence regressors were used to predict the inhalation and exhalation air volume. Several regressors were tested and four of them showed excellent performance: random forest, adaptive boosting, gradient boosting, and decision trees. The latter showed the best regression results, with an R-square value of 0.9998 and a mean square error of 0.0023. The results of this study showed that non-contact spirometry based on a thermal imaging system is feasible and provides all the basic measurements that the conventional spirometers support.

**Keywords:** thermal camera; non-contact spirometry; artificial intelligence regression; respiration signal; respiration rate mobile application

#### **1. Introduction**

Pulmonary diseases such as asthma, chronic obstructive pulmonary disease (COPD), pneumonia, bronchitis, pleural effusion, and lung fibrosis affect tens of millions of people all over the world [1]. The monitoring of respiratory function is of great importance in diagnosing these diseases. One of the most widely used methods of diagnosing and assessing the progression of respiratory diseases is spirometry. In conventional spirometry, the patient wears a mouthpiece for breathing and puts on a nose clip. The patient is instructed to breathe in a certain procedure so that several parameters can be measured during the breathing protocol. The patient is instructed to inhale and exhale into the spirometer mouthpiece at a certain breathing rate and force, then the spirometer calculates the breathing rate and the volumes of inhaled and exhaled air during the measurement procedure. As an example of using the spirometer in clinical procedures to diagnose obstructed airways, the ratio of the forced expiratory volume in one second (FEV1) to the forced vital capacity (FVC) (i.e., FEV1/FVC) is measured, where FEV1 represents the maximum amount of air the subject can exhale in 1 s and FVC represents the greatest volume of air the subject can exhale after a deep inhale [2]. A low FEV1/FVC value means

**Citation:** Fraiwan, L.; Khasawneh, N.; Lweesy, K.; Elbalki, M.; Almarzooqi, A.; Abu Hamra, N. Non-Contact Spirometry Using a Mobile Thermal Camera and AI Regression. *Sensors* **2021**, *21*, 7574. https://doi.org/10.3390/s21227574

Academic Editor: Giacomo Oliveri

Received: 30 September 2021 Accepted: 6 November 2021 Published: 15 November 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

that the amount of air that the airways can quickly exhale out of the lungs is reduced, which indicates that the airways are obstructed [3,4], while an increase in the FEV1/FVC value results in a restrictive condition that affects the ability to inhale. According to the American thoracic society, an FEV1/FVC value of 0.7 or above is considered normal for adults, while a value of 0.85 or above is considered normal for children aged 5 to 18 years [5].

The current techniques that are used for spirometry can be classified into two categories: contact spirometry and non-contact spirometry. Contact spirometry is the traditional and the most common method in clinical practice. This conventional way of performing spirometry can be uncomfortable for patients, since it requires them to keep the spirometer in their mouth for up to 30 min. Moreover, after each patient use, the spirometry mouthpiece is subjected to a sterilization procedure to prevent any cross-infection among the patients. There have been several techniques developed for non-contact spirometry. Most of these techniques depend on body movement detection [6,7], photoplethysmography (PPG) [8], and imaging techniques. Droitcour et al. [6] and Mostov et al. [9] used Doppler radar to measure the movement of the body due to heart and respiration signals. For this, they used a high-quality CMOS Doppler radar sensor and verified their results against standard heart and respiration devices. An image-based system was used by Lin et al. [10], where they developed a method based on harmonic analysis of body motion to detect changes in the respiration rate pattern. The second category of non-contact spirometry techniques is based on signal analysis of the PPG signals. In this category, the change in blood volume in the blood vessels is measured by observing changes in light absorption. The measured PPG signal is normally modulated by the respiratory signal (activity) due to movement of the chest cavity during blood flow. These techniques that are based on the analysis of PPG signals are normally based on signal processing techniques and can provide only frequency information about respiration without any information about the changes in the lung volume. Madhav et al. [8] used a modified method of principal component analysis (PCA) to separate the heart activity from the respiration activity. The recorded PPG signal was acquired by a pulse oximeter. Another study depending on ambient light for the recording of the PPG signal was performed by Verkruysse et al. [11]. They used harmonic analysis for extraction of the different components of the PPG signal including the respiration activity. Al-Naji et al. [12] used an ultrasonic PING sensor and PIC18F45 microcontroller to monitor the normal and abnormal breathing activity of the subjects continuously and instantaneously at different distances. The third category of non-contact spirometry techniques is based on thermal imaging, where measurement of the respiration activity is performed through monitoring the changes in the air temperature during inhalation and exhalation [13,14]. Murthy and Pavlidias used a highly sensitive imaging system in their measurements [14]. They used a statistical model of the recorded image to extract respiratory breathing information. Another study by Murphy et al. [15] evaluated the airflow rate using a thermal imaging system.

Most of the non-contact techniques mentioned in the literature do not provide complete respiratory values as the conventional method does, such as the volume of air inhaled and exhaled during the measurement protocols [6,8–10,14,16,17]. Moreover, most times, they require sophisticated techniques, a special setup, and special hardware, while conventional spirometry is considered simple and cheap, with the only disadvantage of requiring direct contact with the patient, which could produce cross-infection and discomfort. Therefore, any non-contact spirometry system should address these issues. This present work provides a feasibility study of using a mobile thermal camera to replace the conventional spirometer for measurements of the respiration rate and the volume of inhaled and exhaled air in and out of the lungs. The purpose of this study was to build a simple non-contact respiratory measurement system using a mobile thermal camera. The targeted measurements included the respiration rate and the volume of inhaled and exhaled air based on artificial intelligence (AI) regressors.

#### **2. Materials and Methods**

The complete structure of the proposed study is shown in Figure 1. A mobile phone, with a thermal camera connected to it, was used to acquire thermal images from the face of the subject using a mobile application that was built specifically for this purpose. The mobile application extracts the respiration signal and exports it to a back-end server for further spirometry analysis of the inhaled and exhaled air volumes, which was performed based on the AI regression approach. The spirometry measurements were carried out using a conventional spirometer.

**Figure 1.** Graphical abstract of the proposed work.

#### *2.1. Thermal Image Acquisition System*

The thermal image acquisition system consisted of a mobile phone (Samsung S6) with an Android platform and a FLIR I thermal camera [18]. The camera produced images with thermal resolution of 160 × 120 pixels with a pixel size of 12 microns. Its accuracy was around 3 ◦C or ±5% of difference between the ambient and the scene temperature. Both still and video thermal images could be recorded, with video images recorded at a rate of 9 frames per second. A special application that used the FLIR I software development kit (SDK) (available at https://developer.ir.com/mobile/ironesdk/, accessed on 10 October 2021) was built for image acquisition. Moreover, the application used the OpenCV android SDK library for image processing. The image acquisition rate was fully controlled by the FLIR I SDK, with a static value of 9 frames per second. The application detected the region of interest (the area under the nose or the mouth) initially using a regular camera image and calculated the average temperature in that region. The application also specified the duration of the recording; a default duration of 20 s was used initially. The recorded signal of temperature versus time was displayed by the application and transmitted to a PC for further spirometry measurements.

#### *2.2. Respiration Rate Measurement*

The mobile application that was built for this study was used for two purposes: the first one was to record and transmit the respiration signal vs. time to a PC for further measurements, and the second one is was calculate the respiration rate from the recorded signal by finding the number of peaks (local maxima) in that signal divided by the duration of the recording. The mobile application indicated whether the respiration rate was normal (15–24 breaths/min) or abnormal. The flow chart of the mobile application is shown in Figure 2.

**Figure 2.** Mobile application workflow diagram.

#### *2.3. Spirometry Measurements*

The spirometry measurement approach adopted in this work was an artificial intelligence system based on regression trees. The system's plan of operation is shown in Figure 3. To be able to build such a system, a conventional spirometer was used to provide data about the change in lung volume during measurements. These measurements provided the ground truth for the training and testing of the AI system. The spirometer used in this study was a Contec SP80 healthcare portable medical patient spirometer (shown in Figure 4) [19]. Data were acquired from different subjects, where each time, the change in lung volume during the respiration protocol was recorded.

**Figure 3.** Plan of the spirometry measurement system.

**Figure 4.** SP80B spirometer, Contec Medical Systems.

#### 2.3.1. Subject

The measurements used in this study were taken from 18 subjects. The age of the subjects ranged from 16 to 28 years, with an average age of 22.5. The subjects' mass was in the range of 54 to 90 kg, with an average value of 67.5 kg, while their height was in the range of 159 to 188 cm, with an average value of 167 cm.

#### 2.3.2. Regression Models

The recorded signals of temperature versus time were used to build a regression model to predict the value of the change in volume during the respiration protocol. Several regression tree models were tested for this work. The workflow of the proposed model is shown in Figure 5.

**Figure 5.** General workflow for the regression procedure.

− Decision Trees Regressor

Decision trees (DT) were first introduced by Breiman et al. in 1984 [20]. This method is widely used due to its simplicity and efficiency. The learner uses a decision tree as a predictive model for the output. The decision tree is constructed based on the recorded signal, which is considered as a feature vector (observation) for training and testing the trees. It basically splits the training dataset into smaller and smaller subsets, while simultaneously developing the corresponding decision tree in an incremental manner. The DT normally works as a top-down scheme, where each observation is used to measure the output based on a certain measure such as Gini impurity, information gain, and variance reduction. To predict a particular data point, the algorithm runs through the complete decision tree by answering "true/false" questions at every node, until arriving at the leaf node which represents the outcome. Most importantly, the same procedure is iterated multiple times to increase the accuracy of prediction. The major advantage of DT is the simplicity and the ability to handle numeric variables (regression), while the major disadvantage is the sensitivity to noise in the observations, which may cause a large variation in the decision trees and hence increase the error in the predictions [21].

− Random Forest Regressor

The random forest classifier and regressor was first introduced by Breiman in 2001 and has been subjected to several improvements since then [20,22]. The random forest is a supervised machine learning technique that deploys multiple trees running in parallel during the training stage with no interaction among the trees. The outputs of all trees are aggregated to calculate the final output of the random forest regressor. If the vector *x* contains *N* features such that *x* = [*x*1, *x*2, ... , *xN*], these features are used to predict the output for a given input feature. The vector *X* in this study represents the thermal respiration signal recorded during the measurement protocol. The signal is used to build a predictive model to predict the value of the respiration volume. If the true measured volume is *V*, then the predicted volume is *V*,.

This algorithm works in the following steps: firstly, it selects uncorrelated and random sub-samples from the training dataset and builds up a separate decision tree for each sub-sample. Next, to predict a particular data point, it extracts a prediction result from every decision tree. Lastly, it applies a vote procedure to all available prediction results and declares the prediction result with the greatest number of votes as the final predication. In random forests regression, the voting is the average prediction of each tree; *V*,*i* is the output for of an individual tree *i*. If there are *M* trees in the random forests regressor, then the final output of the is calculated as:

$$
\overline{V} = \frac{1}{M} \sum\_{i=1}^{M} \overline{V}\_i
$$

Some of the advantages of the random forest regressor include its efficiency in handling large databases, its accuracy in performing classification and regression, and its accuracy in the case of missing data points. The main weakness of the random forest regressor is overfitting in the presence of noisy training data.

### − Gradient and Adaptive Boosting Regressors

Boosting classifiers and regressors start the learning process using weak classifiers and regressors such as decision trees. They modify these weak learners to make them strong learners by trying to improve the initial model. In boosting techniques, multiple individual models are trained in a sequential way, while every successive model keeps on learning from mistakes of its predecessor model. There are several techniques for this procedure, with the most common ones being gradient boosting (GB) and adaptive boosting (AdaBoost) [21,23]. In the gradient boosting technique, several models (decision trees) are trained either in an additive or sequential manner. The learner tries to identify the weaknesses of the weak model using the gradients of the loss functions; the performance measure of the regressor used in this work was the mean square error. The different models are updated on the basis of the gradient of the loss function, and weak learners are improved. The gradient boosting regressor works as follows: firstly, it trains a decision tree with an original training dataset and extracts predictions from the trained decision tree. It then computes the residual error of the trained decision tree and stores its value as a variable *Y*. To keep on learning from previous mistakes, the next stage will utilize the calculated residual error of each stage. It repeats the same procedure for a prespecified number of trees to train the model. To predict a particular data point, it simply adds up the prediction results of all trees. In the adaptive boosting technique, the learner identifies the weak classifier (trees) using the cost function, modifies them to enhance their performance, and builds a second regression model based on these analyses. The process continues until an accurate model is completed or a certain number of models has been reached. Similar to the gradient boosting technique, firstly, it trains a simple decision tree with the original training dataset. It computes the weighted error rate (number of false predictions out of total predictions) for the decision tree. It then calculates the weight of the decision tree in the ensemble and updates the weights of all misclassified data points (wrong predictions). Lastly, it repeats the same procedure for a prespecified number of trees to train the model. To predict a particular data point, the adaptive boosting algorithm multiplies the tree's weight by the tree's prediction and adds up all the trees. Consequently, the tree with the highest weighting will be the most significant influencer of the final prediction.

#### 2.3.3. Performance Measures

The performance of the different regressors was evaluated based on four different measures: the coefficient of determination or R-squared (*R*2), the mean square error (MSE), the root means square error (RMSE), and the mean absolute error (MAE). These performance measures were calculated during the regression procedure based on 10-fold cross-validation.

The R-squared value is a statistical measure of how close the predicted data using the regression model are to the true measured value of the spirometer, based on the sum of squared errors [24,25]. It is given by:

$$R^2 = 1 - \frac{SS\_{res}}{SS\_{total}}\tag{1}$$

where *SStotal* and *SSres* are given by:

$$SS\_{total} = \sum\_{i=1}^{n} (y\_i - \overline{y}) \tag{2}$$

$$SS\_{res} = \sum\_{i=1}^{n} (y\_i - \hat{y}\_i) \tag{3}$$

where *yi* is the volume measured using the spirometer (the ground truth), *y* is the average value of the measured values, *y*ˆ*i* is the predicted regression model volume, and *n* is the number of signals tested (sample size).

The *MSE* is a measure of the average error between the measured and the predicted values and is given by:

$$MSE = \frac{1}{n} \sum\_{i=1}^{n} (y\_i - \mathcal{y}\_i) \tag{4}$$

The *RMSE* is the square root of mean square error and is given by:

$$RMSE = \sqrt{\frac{1}{n} \sum\_{i=1}^{n} (y\_i - \hat{y}\_i)} \tag{5}$$

The *MAE* is a measure of the absolute error between the measured and the predicted volumes. It is given by:

$$MAE = \frac{1}{n} \sum\_{i=1}^{n} |y\_i - \hat{y}\_i| \tag{6}$$

#### **3. Results**

The complete proposed system was implemented using a mobile application with the functionality illustrated in Figure 6. The system can provide two types of measurements: the first one is the respiration rate and the second one is the respiration volume measurement, which depends on the measurement protocol. Three volume measurements were programmed in the app: the normal respiration volume (tidal volume), the functional reserve capacity (FVC) or deep breathing, and the FEV1/FVC ratio.

#### *3.1. Respiration Signal and Respiration Rate Measurements*

This module was implemented as a mobile app with a workflow described in Figure 2. The application was programmed using Java programming language in the environment of the Android studio. Thermal image acquisition was performed using the mobile thermal camera software development kit, and facial recognition and the area of interest were implemented using the OpenCV image processing library. Figure 7 shows the interface of the mobile application with the respiration rate measurement and the respiration signal. The two measurements were recorded at a distance of around 20 cm from the subject's face with a duration of 20 s as a default value. The recording was carried out at room temperature (23 ◦C).

The recorded respiration signal was exported to a back-end server for further measurements. Figure 8 shows an example of these recordings at three different respiration rates.

**Figure 6.** Mobile application functionality of the proposed system.

**Figure 8.** Respiration signal recorded at different rates: (**a**) normal breathing rate, (**b**) low breathing rate, and (**c**) high breathing rate.

To explore the effect of the camera distance from the subject's head, four respiration signals were recorded at 10 cm, 30 cm, 60 cm, and 100 cm, as shown in Figure 9.

**Figure 9.** Respiration signals recorded at 10 cm, 30 cm, 60 cm, and 100 cm.

#### *3.2. Spirometry Measurement System*

The spirometry measurement system was completely performed in the Python 3.7 programming environment. The package Scikit-learn 0.24.1 was used for this purpose [25]. The four regression models mentioned earlier were considered. The regression testing and training were carried out using a 10-fold cross-validation technique. The performance of the regression of four techniques is listed in Table 1. The performance measures indicated that the decision trees, along with the gradient boosting regression methods, had better

performance in comparison with the other two methods, with slightly superior performance for the decision trees.

**Table 1.** Summary of the regression methods used.

Testing of the proposed methods was carried out using a 10-fold cross-validation procedure. Figure 10 shows the agreement between the predicted change in volume of the method in comparison with the measured spirometer value. A perfect agreement is indicated by a line of a slope of 1. The performance of the decision trees method is evident in Figure 8c and the performance measures in Table 1.

**Figure 10.** Results of the 10-fold cross-validation testing procedure: (**a**) random forests regressor, (**b**) gradient boost regressor, (**c**) decision trees regressor, and (**d**) adaptive boosting regressor.

#### *3.3. Spirometry System Testing*

The complete proposed system was tested for volume measurements with several breathing volumes and compared with the reading provided by the conventional spirometer. The first test was performed on a normal subject with a normal breathing rhythm. As shown in Figure 11, a breathing volume of 0.55 L was recorded using the mobile spirometer

application, while the conventional spirometer recorded a breathing volume of 0.59 L. Another breathing test was performed to measure the FVC of a normal subject. The proposed system showed a measured volume of 3.01 L, which was exactly the same volume measured by the conventional spirometer. The measurements are shown in Figure 12.

**Figure 11.** Measured normal breathing volume (tidal volume) (**a**) using the mobile application with a thermal camera and (**b**) using the conventional spirometer.

**Figure 12.** FVC test measurement: (**a**) mobile app reading; (**b**) conventional spirometer reading.

The last measurement test was the FEV1/FVC ratio. The FVC was measured as mentioned previously, while the FEV1 was measured based on the change in respiration volume with time (1 s). Figure 13 shows the FEV1/FVC measurement using both the mobile app and the Contec spirometer.

**Figure 13.** FEV1/FVC measurement (**a**) using the mobile app and (**b**) using the Contec spirometer.

#### **4. Discussion**

This work presents a feasibility study for respiratory measurements using a mobile thermal camera. The measurements were made by a first-generation FLIR I mobile thermal camera. Although there are more advanced thermal cameras now, due to the need for temperature monitoring because of the COVID-19 pandemic, the camera provided good results for respiration rate measurements at a close distance. The FLIR I is a very basic first-generation mobile thermal camera with low resolution compared with other advanced cameras. The camera provides the highest possible accuracy by automatically calibrating itself according to the scene temperature by calculating the emissivity of the objects, the reflected temperature, and the distance; it then returns the actual temperature of the object. The accuracy of the recorded temperature was reported in the literature to be around 3 ◦C [26]. The effect of the measurement accuracy was minimized in this study for two reasons: first, we measured the average temperature of the area of interest and not pixel by pixel temperature, which resulted in reducing errors in the measured temperature. The second reason is that the measured signal represents the change in temperature and not the actual temperature, which cancels the effect of errors that may appear during the calibration of the camera.

In the current study, the distance was around 20 cm; greater distances resulted in degradation of the signal quality, as shown in Figure 6. Therefore, higher-quality cameras may result in a better respiration signal and thus better measurements at higher distances. One of the most promising applications of respiratory rate measurements is sleep apnea (cessation of breathing). The mobile thermal camera, along with the mobile application, could provide a non-contact and hassle-free system to detect apnea during sleep, as the current clinical apnea monitoring system requires wiring to the patient's body.

The FLIR I thermal camera has an image acquisition rate of nine frames per second; therefore, the average temperature has a sampling rate of 9 Hz. The respiration rate varies with age and the maximum breathing rate can reach up to 55 breaths per minutes in neonates [27]. This means that the maximum breathing frequency is 1.1 Hz and therefore, the current image acquisition rate of the FLIR I is enough to provide a faithful representation of the respiration waveform. At the same time, higher imaging rates would provide a better representation of the respiration signal and hence better performance.

The change in the lung volume (spirometry) measurement was performed using an AI approach based on different regression models. The decision trees regressor showed excellent performance in predicting the change in volume due to respiration, with a mean R-square value of 0.9998 and a mean square error of 0.0023, as listed in Table 1. Moreover, the AI system was built (training and testing) using a range of volumes (0.2 L to 3.5 L), which makes it appropriate for a wide range of subjects and ages. Several studies have adopted a non-contact approach for respiration measurements [8,28,29]. Brieva et al. [30] estimated the respiratory rate using a Hermite magnification technique and a convolutional neural network (CNN). Their contactless system monitored the subjects' chest movements with and without an ROI (region of interest), and they evaluated the performance of their methods using the mean average error. With and without an ROI, the respiratory rate was estimated with a mean average error of 1.83 ± 1.61% and 3.28 ± 3.33%, respectively. Liu et al. [31] proposed an imaging-based approach to determine spirometry parameters such as FEV1, FVC, and PEF. They captured images of the subjects' faces and shoulders during inspiration and expiration using a webcam. Their method showed fair performance, with a root mean square error of 0.27, 0.18, and 0.56 and an average error of 8.5%, 6.9%, and 7.7% for FEV1, FVC, and PEF, respectively. However, using infrared thermography, Pereira et al. [32] estimated the breathing rate with a mean absolute error of 0.33, 0.55, and 0.96 bpm (breaths per minute). One of the most recent studies is the study by Schoun et al. [28]. They used an in-vitro benchtop thermal imaging setup to acquire simulated measurements. They used a recurrent network model (AI) to predict the respiration signal, and from the predicted signal, they made respiration measurements. Using RMSE, their LSTM neural network predicted the expiratory volume from human tests with an average error of 10.61%. However, in this study, the change in volume due to respiration was predicted directly using AI regression models.

#### **5. Conclusions and Future Work**

The current study presents a successful approach for respiratory measurements of the respiration rate and volume change. It could provide a means for a non-contact, continuous, and accurate method for the monitoring of chronic and acute pulmonary diseases. However, more steps forward are still needed to improve the quality of the thermal signal measurements by using a more advanced thermal camera and probably more advanced image processing techniques for detection of the area of interest in the face. Advanced thermal cameras can provide better signal quality at longer distances and, at the same time, better image quality. Moreover, an increase in the image acquisition rate should significantly improve the proposed system. This study could also be further developed to be used for simultaneous multiple subject monitoring and measurements, especially with the current COVID-19 pandemic. This study could also be tested in a clinical setup with diseases such as asthma, chronic obstructive pulmonary disease (COPD), and pneumonia. Moreover, this study could be performed against different temperature background temperatures to check the reliability of the measurements in a neutral thermal zone. This will require a special setup that can produce different thermal background temperatures.

**Author Contributions:** Conceptualization, L.F.; methodology, L.F., K.L. and N.K.; software, N.K., M.E., A.A. and N.A.H.; validation, L.F.; formal analysis, L.F. and N.K.; investigation, L.F., M.E., A.A. and N.A.H.; resources, L.F. and N.K.; writing—original draft preparation, L.F.; writing—review and editing, M.E. and K.L.; visualization, L.F. and N.K.; supervision, L.F.; project administration, L.F.; funding acquisition, L.F. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by Abu Dhabi University research office.

**Institutional Review Board Statement:** The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Institutional Review Board of Abu Dhabi University (File Number: CoE-21-06-019, 27 June 2021).

**Informed Consent Statement:** Informed consent was obtained from all subjects involved in the study.

**Acknowledgments:** The authors would like to thank Eng. Tasnim Basmaji for her help and support during the entire period of the project.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

### *Article* **Brain Strategy Algorithm for Multiple Object Tracking Based on Merging Semantic Attributes and Appearance Features**

**Mai S. Diab 1,2,\*, Mostafa A. Elhosseini 3,4, Mohamed S. El-Sayed 1 and Hesham A. Ali 3,5**

**Abstract:** The human brain can effortlessly perform vision processes using the visual system, which helps solve multi-object tracking (MOT) problems. However, few algorithms simulate human strategies for solving MOT. Therefore, devising a method that simulates human activity in vision has become a good choice for improving MOT results, especially occlusion. Eight brain strategies have been studied from a cognitive perspective and imitated to build a novel algorithm. Two of these strategies gave our algorithm novel and outstanding results, rescuing saccades and stimulus attributes. First, rescue saccades were imitated by detecting the occlusion state in each frame, representing the critical situation that the human brain saccades toward. Then, stimulus attributes were mimicked by using semantic attributes to reidentify the person in these occlusion states. Our algorithm favourably performs on the MOT17 dataset compared to state-of-the-art trackers. In addition, we created a new dataset of 40,000 images, 190,000 annotations and 4 classes to train the detection model to detect occlusion and semantic attributes. The experimental results demonstrate that our new dataset achieves an outstanding performance on the scaled YOLOv4 detection model by achieving a 0.89 mAP 0.5.

**Keywords:** multiple object tracking; data association; dataset; deep learning; semantic attribute

#### **1. Introduction**

Object tracking is of great interest to researchers because of its numerous computer vision applications such as robot navigation, self-driving, and smart surveillance. It attempts to give a unique ID for every object throughout all frames. However, many fundamental challenges abound regarding the development of a robust object tracking model. These include unpredictable changes in the background and the appearance of objects, full or partial occlusion, and data association. Over the years, researchers have developed robust algorithms to address these challenges. Their methods, however, have not solved one of the main problems in MOT, which is occlusion.

The human brain can identify both stationary and dynamic objects. Moreover, it can effortlessly perform vision processes, including activity recognition, image compression, and motion analysis. Therefore, we believe that the uncertainty in the tracking problem could be solved by developing an object tracking algorithm that can mimic the human brain. Merging researchers' efforts in the neuroscientific community who have studied the approaches used by the human brain to track objects in real-time and researchers' attempts in the computer vision community are needed to improve MOT models. This is what we have achieved in this paper.

**Citation:** Diab, M.S.; Elhosseini, M.A.; El-Sayed, M.S.; Ali, H.A. Brain Strategy Algorithm for Multiple Object Tracking Based on Merging Semantic Attributes and Appearance Features. *Sensors* **2021**, *21*, 7604. https://doi.org/10.3390/s21227604

Academic Editors: Ayman El-baz, Guruprasad A. Giridharan, Ahmed Shalaby, Ali H. Mahmoud and Mohammed Ghazal

Received: 8 October 2021 Accepted: 11 November 2021 Published: 16 November 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

In neuropsychology, the cognitive function of recognising people includes simultaneous routes of information and multiple processing stages. Models of these multiple processing stages have been provided from neuropsychological data obtained from case reports. According to research on young adults [1], humans can simultaneously recognise objects and extract meaning from what they see in the first half-second—but what part of the brain is responsible for this? Researchers in neuroscience have used advanced techniques to identify the brain areas that are involved in object tracking, for instance, magnetoencephalography (MEG), whole-brain functional magnetic resonance imaging (fMRI), and electrocorticography (ECoG) [2,3]. It has been found that this ability involves continuous activity within the anteromedial temporal cortex through the ventral temporal cortex [3]. While visual input is processed through this pathway, it has been found that it is transformed into an initial semantic representation (plants, animals) [4]. This happens in the inferior temporal cortex, before specific semantic representation emerges (rose, cat) in the anteromedial temporal cortex. Another conclusion drawn by [4] is that people recognition includes many attributes, such as the perception of faces, whole bodies, emotional expressions, clothes, gait, and individual marks. In other words, with these biological processors, humans can transform object representations or visual inputs into a format that is easily understood and extractable, regardless of the viewing conditions; this is the semantic attribute. In our algorithm, we aimed to design a tracker that mimics the working principle of the brain to enhance the performance of computer vision state-of-the-art algorithms. Our tracker is motivated by contemporary cognitive psychology. It combines an appearance feature vector (inspired from computer vision algorithms) and a semantic attribute (inspired by cognitive psychology) to represent the object. The appearance feature vector is used to track targets with no occlusion, while the semantic attributes are used to prevent incorrect tracking of an occluded object.

This research proposes novel brain-based MOT algorithms based on eight human brain concepts. Our approach, which resolves the frame-by-frame association problem, is novel; it combines appearance feature vectors and several semantic attributes (trousers, shirts, men, and women) to recognise objects. The main contributions are three-fold:

The remainder of this paper is organised as follows: the next section explores various previous brain-inspired MOT algorithms. Section 3 presents the proposed method in detail. A quantitative evaluation for our algorithm (Merging Semantic Attributes and Appearance Feature MSA-AF), compared to five state-of-the-art algorithms, has been introduced in Section 4. Section 5 shows our algorithm's limitations, while Section 6 presents what went wrong in our dataset. Section 7 concludes.

#### **2. Related Work**

Multiple object tracking has long been a widespread topic in computer vision, with a large number of publications. Over the last few years, researchers have put forward several brain-inspired tracking algorithms. However, these algorithms can only adopt one of the key cognitive effects used by the human brain, such as memory and attention, instead of imitating the entire process [5,6]. Other algorithms have integrated powerful

tools that mimic the human nervous system to tackle the occlusion problem, such as deep neural networks (DNNs) [7], artificial neural networks (ANNs) [8], recurrent neural networks (RNNs) [9], and convolutional neural networks (CNNs) [10]. These networks can simulate the processes used by the human brain to identify, store, or process information. However, according to the research findings of [11], ANNs are fundamentally different from biological neural networks (BNNs) in many ways. For instance, cortical neurons in BNN and artificial neurons in ANN communicate to each other differently. BNN's neurons are organised in a goal-driven manner, unlike ANN's neurons. Additionally, training is not needed in BNNs, unlike ANNs, in which the weight of the neural connections needs to be trained to obtain better results.

The question that emerges here is how can we judge the degree to which a model agrees with biological outcomes? Despite no definite answer to this question in the literature, we will try to find answers using findings from the cognitive neuroscience of MOT and what these computer vision trackers have introduced in their models.

Researchers in cognitive science and neuroscience have executed many studies to examine the cognitive mechanism behind MOT. According to their outcomes, MOT tasks focus on visual attention in early cognitive processing, while in later cognitive processing, visual short-term and working memory have been used [12]. Thus, we categorised the computer vision algorithms that imitate brain functions in their designs into two categories: memory-based algorithms and attention-based algorithms. We will cite a few MOT algorithms based on visual attention and short and long memory without being exhaustive.

Visual attention helps humans analyse complex scenes promptly and dedicate their limited cognitive and perceptual resources to the most relevant subsets of sensory data. Some human brain-inspired algorithms that achieved high accuracy predicted what part humans will attend to in an unobserved image. One of the best models for predicting fixation over images and videos is the decision theoretical model [13], which has shown promising results on the PASCAL 2006 dataset using the discriminant saliency model for visual recognition. This model has explained basic behavioural data that other models have explored less. In computer vision, attention is a convenient method for recognising targets by discriminating them from each other and the background. In [14], dual matching attention networks (DMAN) with a temporal and spatial attention mechanism strategy is proposed to allow the model to focus on matching similarity between each part of a pair of images.

The temporal attention network aims to investigate different observations in the trajectory by allocating different degrees of attention to them. The spatial attention module allows the network to focus on the matching patterns. Unlike [14], in which their attention model relies on only two aspects, [15] the temporal attention network focuses on four attention aspects to track the target. Two of these attention aspects are common with [14], which are temporal and spatial attention. The other two are layer-wise and channel-wise attention. However, the limitation of [15] is that their algorithm needs offline training to support feature selection in online tracking. Even though attention helps to tackle challenges such as background variability and variable target appearance, it suffers from occlusion due to its adaptability to variation. When a target is gradually occluded by an occluder, the algorithm misses the target because it tends to acknowledge it as a change in target appearance. Memory could be used to handle the occlusion by retaining the appearance information from the targets. [16] implement their network with long shortterm memory (LSTM) units that improve their algorithm's performance.

Visual memory is an essential cognitive mechanism that the human brain uses to deal with partial and full occlusion and to overcome variable target appearances. Makovski and Jiang [17] proved in their studies that the reason behind the observer's recovery from any errors during tracking was the storing of surface properties inside the visual working memory. In [12], they propose a general theoretical framework for viewing human memory. They divide the memory model into three stages: the sensory register, the short-term store, and the long-term store. Many researchers in computer vision have used this theory to

introduce brain-memory-inspired MOT [18,19]. In [18], they established a memory model to handle the main challenges in MOT by using the human brain three-stage memory theory [12]. Using this model, they proposed a template updating modelling algorithm that can remember and forget the target's appearance. Their experimental results are promising for tackling sudden changes in appearance and occlusion. In [19], they are inspired by the same three-stage memory model. However, the three-stage memory model in this work aimed to find the right match between the appearance and the object by integrating the model into a multiagent coevolutionary process. Each agent can forget and remember the object's appearance via its own experience through its memory system. Other researchers have used different strategies employing long short-term memory (LSTM) instead, such as [20–22]. Ref [20], introduced an algorithm based on deep reinforcement learning followed by the LSTM unit. Their track-by-detection algorithm detects the object and then applies an association model to solve the tracking problem. LSTM is used in the data association stage to address changes in appearance and occlusion. The experimental results of [20] show that their tracker is successful in handling scale changes, occlusion, and appearance similarity most of the time. However, their tracker faced some failures in some cases. For example, when the brightness of the environment changes, the detector fails to detect the object, leading to the tracker's failure.

All algorithms introduced in this section are based only on attention or memory modules, added to their algorithm to imitate the human visual role. Although attention is undoubtedly involved in feature extraction and memory is involved in retaining extracted information, they are rarely used together in the same algorithm. In MSA-AF, we used not only both modules, but also six other strategies inspired by the human brain. We introduced the advantages and limitations of both models, memory and attention, in Table 1.

**Table 1.** Human brain-inspired algorithms.

#### **3. Proposed Algorithm**

In this section, our tracking algorithm has been presented, which uses the outcomes from the neuroscience researchers' community to imitate the human brain. What is critical in our work is to translate these biological findings into a computer vision algorithm. Both the neuroscience and computer vision communities share a general framework that focuses on the importance of memory, attention, and motion prediction to handle the MOT problem. To the best of our knowledge, no algorithms use all eight of the human brain strategies introduced in Table 2 as their building blocks. In contrast, our work was inspired by eight main findings from neuroscience research as building blocks to build MSA-AF. Table 2 shows these modules and how they inspired our algorithm. Integrating the eight modules together is undoubtedly ambitious, leading to improvements in the state-of-the-art MOT algorithms. However, strategies 4 and 8 from Table 2 played the most critical role in the final association stage, as discussed in more detail later on in this section.

**Table 2.**

The basic flow chart of the tracking algorithm we implemented in this work is illustrated in Figure 1. Our focus was on handling the existing challenges facing state-of-the-art algorithms, the most important of which was occlusion. The first building block in our algorithm is detection, which takes a frame and gives us the bounding boxes (Bbox) of the four classes (man, woman, shirt, trouser) inside the image. Then, the second block will take the Bbox from the previous frame, apply the Kalman filter [26] to predict their new position in the current frame, and then apply the cosine metric learning method for learning a feature space to associate the detection with the objects. The last three building blocks distinguish MSA-AF from any other MOT in the field of computer vision, which is similar to how human logic works to solve a tracking problem. This occurred by transferring the information in the image into semantic information that is saved in the memory for retrieval when it is needed. This brings us to the last building block, which uses the saved semantic information to decide whether occlusion is happening or not and how to deal with it. The rest of this section will introduce the logic behind each building block and how all are connected to imitate the human brain and improve the performance of recent MOT algorithms.

**Figure 1.** Overview of MSA-AF starting from the detection step, which feeds into the first association step. The final association step will be used in cases of occlusion. Track 1 to Track n represents the model of the object that has been tracked in the previous frames. A class could be 0, 1, 2, or 3, representing man, woman, shirt, and trousers. Bbox has x,y,h,w of the object's bounding box, in which (x,y) represents the centre point, h is the height, and w is the width. The occlusion state is a binary number: '1' if the object is in an occlusion state and '0' if not. Occluded with has the ID of the object that was occluded. Age is the number of frames in which the object is successfully tracked. Finally, Semantic information includes much information that will be described in detail in Section 3. The original image is taken from the MOTChalleng dataset [28].

#### *3.1. Detection*

Without exaggerating, we can say that the detection step is essential for any trackingby-detection algorithm such as ours. Missing, false, or non-accurate detection will lead to poor MOT performance; consequently, more time is spent on this stage to secure a good foundation for our algorithm. Some algorithms assume that they have the detections needed to focus on the tracking step, such as in [24]. Three crucial questions need to be answered to deal with the detection step successfully: What classes do we need to detect? What is the detection model that will be used? Do we have a good dataset for the training? All these questions will be answered in this section. Figure 2 depicts the different phases used to ensure a reliable output from the detection stage.

**Figure 2.** Steps that have been used to assert a high-performance detection model. Each phase included all options that we used for the comparison in order to get the best detection results.

#### 3.1.1. Classes

In any MOT algorithm that deals with people, the person is the class that needs to be detected. In MSA-AF, as we imitate the human brain, we use a different approach. Biological evidence shows that the human brain transforms the visual input into an initial and then a specific semantic attribute [4]. We employ the person reidentification approach used in multi-camera networks to find a semantic attribute representing a person. This approach identifies the best attributes for recognising a person and helps an operator to track, search, and locate the target object. For example, in the Van Koppen and Lochun [29] survey, over 1313 human descriptions were collected from individuals who witnessed a robbery incident. They found that of the 43 categories identified, only 30% of the eyewitnesses accurately described all nine categories. The categories included skin colour, hair type, height, gender, appearance (which includes race), accent, hair colour, build, and age.

Similarly, Sporer [30] examined 139 descriptions provided by 100 witnesses. He discovered that of all the descriptions, 31% mentioned clothing information, 29.6% detailed facial features, 22% contained information on movement features and physical attributes (height, race, and age), and 5% included personality information. After considering the findings of these two studies, we selected clothes (trousers and shirt) and gender (man and women) as our semantic attributes. We will reserve the term semantic information units to refer to the trouser, shirt, man, woman semantic attributes.

#### 3.1.2. Dataset

The training dataset can be defined as the initial data used to develop the machine learning model. A high-quality training dataset, without a doubt, is the critical item for any machine learning system. Most performance detection models can be rendered useless without high-quality and quantity training datasets.

After deciding what classes we needed to detect, we searched for a dataset to train the detection model with the same classes. However, we could not find any dataset with shirt, trouser, man, and woman classes that could benefit our algorithm. Thus, we built the PGC dataset not only to serve our algorithm, but also to benefit other researchers.