Optical transmitters and optical receivers play an essential role in optical sensing technology. In this article, we introduce a low-cost method to use this technology. The lightemitting element and the photosensitive element are obtained by modifying the commercial LED strip (Telesky, Shenzhen, China), as shown in the figure. The LED strip is based on a flexible printed circuit board (FPC), a photodiode powered by 5 V and a corresponding current limiting resistor. Each light-emitting diode is independently powered and can normally work when the anode and cathode are connected to a 5 V power supply. It is worth noting that the applicable model of the photodiode is 5050 (5 mm × 5 mm), which means that the LED footprint can fully accommodate the 1206 SMD surface mount package. Therefore, according to the positive and negative polarity, we chose a 1206 SMD surface mount package type photoresistor, replacing the lamp beads in the LED light strip. At the same time, we soldered a signal wire from the voltage divider circuit node and led it out. After using a jumper wire to connect the LED light bar and the photoresistor bar, we used hot melt glue to cover the soldering point to improve its reliability, as shown in Figure 2.

**Figure 2.** Fabrication of flexible circuit board inside the sensing unit. (**A**) cut out a single LED unit from the light strip; (**B**) replace the LED lamp beads with a photoresistor; (**C**) weld the power supply and signal wires; (**D**) weld the LED strip and the photoresistor strip together.

#### (b) Flexible light-transmitting medium

The pressure sensing unit needs to tolerate a specific pressure range and return to the original state when the applied pressure is released. As an inexpensive material, organic silicon materials are widely used in the design of many flexible sensors. This article uses semi-transparent silica gel (Beijing Hibas Technology Co., Ltd., Beijing, China) as the primary elastic material and is mixed with corresponding plasticizers to catalyze the solidification process of silica gel. In the initial prototype design, we found that only silica gel was used to prepare a light guide medium with higher hardness, which made the signal not obvious enough for us in the plantar pressure range of interest. To optimize the design of the sensor unit, we have included a softener (dimethazone). By mixing different proportions of silica gel and softener, we finally determined the appropriate ratio of the mixture as the elastomer medium, and its weight ratio is silica gel: softener = 4:1.

#### (c) Integration process

After preparing the circuit and the elastic medium, we used Autodesk Fusion 360 modelling software to design multiple molds for casting and used FDM3D printers to prepare the molds. As shown in Figure 3, one of the molds is used to make the sensor unit's silicone shell baffle (thickness 1 mm). The other mold is used to integrate the entire sensor unit (including the silicone shell baffle, circuit and elastic medium). Figure 3B shows the process of fixing the sensing unit circuit in the grooves of the two silicone shell baffles and, finally, integrating it into a whole with the cured silicone medium. The final size of the sensor is a square flexible sensing unit which is 20 mm in width, 20 mm in length and 7 mm in height. One single sensing unit weighs 2~2.2 × 10−3 kg (silica gel density 700 kg/m3). Figure 4 shows the manufactured sensor with and without power supply status.

**Figure 3.** The integration process of the flexible module of the sensing unit. (**A**) Silica gel baffle pouring (**B**) The sensor unit is fixed on the silica gel baffle (**C**) Put into the mold and add the silica gel mixture to wait for solidification.

(**a**) (**b**)

**Figure 4.** The manufactured sensing unit. (**a**) unit without power (**b**) unit with power.

#### 2.1.3. Characteristic Analysis System

We analyzed the mechanical and electrical characteristics of the manufactured modular sensing unit and calibrated the mapping relationship between the quasi-static pressure and the output signal. It is necessary to conduct a characteristic evaluation experiment on the sensing unit before being integrated into the insole. In the paper [19], the author provides a low-cost calibration method that allows researchers to carry out the calibration test of pressure-related sensing units without using expensive calibration equipment. Here, we introduce an improved version. The calibration analysis system is shown in Figure 5. First, we established a calibration instrument according to the process described in [19] to measure the load force and deformation during the static load test. The calibration instrument is a microsystem composed of three parts: (1) HX711 force measurement unit, on both sides of which are bolted 3D printed rigid plastic (PLA) boards; (2) HX711 amplifier circuit module; (3) an Arduino NANO microcontroller for collecting and recording data from the measuring instrument; secondly, we replaced the original printing platform of the FDM3D printer with a load cell. At the same time, the print head of the printer was replaced with a corresponding contact pressure head according to different test purposes. The loading and unloading can be designed by writing G-code control codes for the 3D printer test. In other words, compared to manually adding weight, the mechanical frame of the 3D printer makes the process more controllable.

**Figure 5.** Characteristic analysis system.

#### *2.2. Insole Solution*

#### 2.2.1. The Layout of Insole

This part mainly introduces the design and production of flexible pressure sensing insoles based on the photoelectric pressure sensing unit designed above. The most challenging problem is that the layout of the sensing unit in the pressure sensing insole needs to fully consider the plantar pressure distribution. From the intuitive impression, due to the irregular surface of the sole and the dynamically changing contact position, the pressure is not evenly distributed on all the surfaces of the insole. For example, the pressure on the inside of the foot arch is slight, while the heel and forefoot areas have greater pressure. Figure 6 is a diagram of plantar pressure distribution in the standing state from reference [13]. From the heat diagram, it can be observed that the plantar pressure is mainly distributed in the heel, forefoot and toes, among which the force in the toe area is mainly located on the thumb. Therefore, placing sensors in these locations can provide more relevant data on plantar pressure.

**Figure 6.** The plantar pressure distribution. (**a**) typical plantar pressure distribution in a standing position; (**b**) first insole layout solution; (**c**) second insole layout solution.

Because the difference in the foot size varies among different people, and the layout of the pressure-sensing insole does not have a proper guideline, two preliminary steps were made based on the author's foot size. A flexible pressure insole solution, as shown in the figure: the layout of the sensor mainly refers to the aforementioned plantar pressure distribution. In the first solution, sensors are placed in six places: the first toe, the third toe, the first metatarsal, the fifth metatarsal, the outside of the arch of the foot and the heel. In the second solution, sensors are placed in six places: the first toe, the first metatarsal, the fifth metatarsal, two outside of the foot's arch and the heel. "S1" in Figure 6b, c represents sensor 1, "S2" represents sensor 2, and so on.

#### 2.2.2. Insole Manufacturing

The manufacturing process of the insole is as follows: first, we designed and 3D printed the casting mold for the insole (right foot), which is size 43 according to Chinese standards; secondly, the sensor unit was fixed in the insole casting mold according to the corresponding position of the two insole layout solutions (six sensors for each solution); all the wires are guided to the outlet at the heel of the casting mold and fixed in the free position where the sensor does not interfere with each other; after the mold outlet is closed with a 3D printed lid, the silica gel mixed with the same proportion softener (dimethazone) was poured into the casting mold—it should be noted that the cavity height of the insole casing mold is 0.5 mm higher than the height of the single sensing unit which ensures the surface of the insole after casting is as flat as possible; finally, the mixed silica gel takes 4 h to solidify, and all the power supply wires are welded into a bus for external power supply. The weight of the two insole solutions are 136 g and 132 g, respectively. Figure 7

shows the two plantar pressure-sensing insole solutions in the non-powered state and the powered state.

**Figure 7.** Two insole solutions for the right foot insole. (**a**) Insole solution No. 1 without power (**b**) Insole solution No. 1 with power (**c**) Insole solution No. 2 without power (**d**) Insole solution No. 2 with power.

#### 2.2.3. Electrical System

To measure the sensor's signal and record the data, we designed the circuit system according to the system framework shown in Figure 8. The circuit system is mainly used for sensor signal acquisition, data preprocessing and data storage, including a microcontroller module, a data storage module and a power supply device. Since the voltage divider circuit of the sensing unit has been integrated inside the sensing unit, there is no need to use additional modulation circuits and operational amplifiers to process the signal. The signal channels from the pressure sensing insole (6 per foot, 12 on both feet) are connected to the input port of the 16-channel multiplexer module (HC4067, NXP). Under the control of the microcontroller (Arduino UNO, Ivrea, Italy), the multiplexer traverses all the connection channels in turn, and transmits the collected analogue signal to the analogue input port of the microcontroller and passes the built-in ADC (analogue-to-digital converter), which converts the voltage signal into a digital signal for storage. The sensor signal data is recorded in a file on the SD card for offline analysis and evaluation of the performance of the plantar pressure-sensitive insole. To improve the overall ease of use, we designed an Arduino UNO expansion integrated circuit board to integrate all the above modules into the expansion circuit board, as shown in Figure 8. The entire system can be powered by a DC voltage source of 5 V~12 V for power supply, such as a polymer lithium battery.

**Figure 8.** Circuit system diagram (**a**) and Arduino UNO expansion integrated circuit board (**b**).

#### 2.2.4. Plantar Pressure Center

COP is widely used in the study of plantar pressure-related parameters, especially the division of the gait phase, so it can be used as the most direct evaluation parameter for verifying pressure-sensing insoles. During exercise, due to the movement of the body's center of gravity, the plantar pressure center shows a periodic trend. It moves from the heel to the toe in a single foot and switches back and forth between the two feet. Therefore, based on our design, we refer to the method introduced in the paper [11] to calculate the COP. The COP is divided into *CoPx* along the inner and outer directions and *CoPy* in the front and rear directions. The calculation method is shown in Equations (1) and (2):

$$CoP\_{\mathbf{x}} = \frac{\sum\_{i=1}^{6} X\_i \cdot P\_i}{\sum\_{n=1}^{6} P\_i},\tag{1}$$

$$CoP\_y = \frac{\sum\_{i=1}^{6} Y\_i \cdot P\_i}{\sum\_{u=1}^{6} P\_i} \tag{2}$$

where *Xi* and *Yi* represent the position of the sensing unit along with the medial/lateral directions and front/rear direction, respectively, as shown in Figure 7. *Pi* represents the signal value of the *i*th sensing unit. It is worth noticing that the plantar pressure center only exists in the standing stage of the leg. Therefore, we define that the center of pressure during the swing stage is located at (0,0) to distinguish the standing and swing phases.

#### **3. Experiments and Results**

#### *3.1. Sensor Characteristic*

Using the calibration analysis system introduced above, we can easily carry out the characteristic analysis experiment of a single sensor. The characteristic analysis experiment is mainly to anchor the center of the sensor unit perpendicular to the static load test of the pressure sensing surface. The static load test is defined as a step of 0.025 mm perpendicular to the sensor's surface and then staying for 3 s to have enough time for stable measurement. The maximum distance is 1 mm (accounting for 14.3% of the thickness of the sensing unit). After the loading process, the unloading process is completed according to the same stepping distance and dwell time until the indenter leaves the surface of the sensing unit. Results of stiffness (force-strain response), sensitivity (resistance-force response) and hysteresis characteristics of a batch of six sensing elements are analyzed.

As shown in Figure 9a, all sensing units exhibit certain mechanical hysteresis characteristics in terms of mechanical characteristics. According to the quantification method of mechanical hysteresis characteristics in the paper [5], that is, through calculation, the ratio of the area enclosed by the loading range and the horizontal axis to the area enclosed by the unloading range and the horizontal axis in the curve is used to quantify the mechanical hysteresis characteristics. The mechanical hysteresis coefficients of each sensor are 0.928, 0.937, 0.947, 0.935, 0.921 and 0.933, respectively. We can observe that the mechanical characteristics of the pressure sensing unit are relatively consistent, which are mainly related to the characteristics of the silicone elastomer inside the sensing unit.

As shown in Figure 9b, there is a certain linear relationship between the sensing signal and the load in terms of electrical characteristics. With the help of MATLAB's cftool toolbox, we chose to use a polynomial to fit the curve. Here, the relationship curve between signal response and loading force is fitted with a two-order polynomial (F(s) = *a*0 + *a*1*s* + *a*2*s*2 where *S* represents the loading force, and F represents the output response signal). The fitting results are shown in Table 1. It can be found from Figure 10 that the electrical hysteresis characteristic is almost negligible during loading and unloading. Combining Figure 10 and Table 1, we can observe that the initial sensing signals (*a*0) of the six sensing units under no-load are more or less different. However, from the 0-60N load range result, the sensor's sensing range is relatively close.

**Figure 9.** The mechanical and electrical characteristics. (**a**) The mechanical characteristics (**b**) The electrical characteristics.

**Table 1.** Fitting Result of Sensing Unit Electrical Characteristic.

**Figure 10.** Signal Curve of the first insole solution.

#### *3.2. Gait Data Collection*

#### 3.2.1. Comparison of Insole Solutions

We evaluate the performance of the two designed induction insole solutions. Preliminary test experiments were carried out on both. The author of this article (male, age 24, height 1.79 m, weight 76.6 kg, shoe size 43) wears two kinds of insoles in indoor corridors according to walking habits, and conducts the following tests: from a natural standing state to a waking state with an average pace, then staying still for a few seconds in the end.

In the experiment, we found that the induction signal has an abnormally negative value. Through observation, we found that the abnormal phenomenon is mainly caused by the asymmetry of the flexible circuit board area in the sensor unit, which causes the internal optical path of the sensor unit to shift when pressure is applied to the circuit around the sensor unit. At this time, the optical path is deflected. The most direct effect of the shift is that the light intensity is reduced compared to the case where the light path is directly facing, which leads to abnormal negative values of the induced signal in the experimental results. Therefore, this is an inevitable feature in the design principle of light-sensitive pressure sensing based on commercial LED light strips in this study. That is, the sensing signal at the center of the sensing unit will be interfered with by the surrounding pressure, but it will be affected by the center of the sensing unit. When the pressure is positive, the pressure signal is in line with theoretical expectations. In response to this phenomenon, we used the linear rectification activation function (Rectified Linear Unit, ReLU) in the neural network to preprocess the signal of the sensing unit and use the negative signal caused by the pressure around the sensing unit under the function of the function. It is filtered, and only the positive part of the sensing signal is retained. The expression equation can be described as:

$$f(\mathbf{x}) = \begin{cases} \quad \text{ } & \text{if } (\mathbf{x} \le \mathbf{0})\\ \quad \text{ } & \text{ } & if(\mathbf{x} > \mathbf{0}) \end{cases} \tag{3}$$

After the rectification activation function is processed, as shown in Figures 10 and 11, the signal curves are drawn from a piece of data intercepted from the walking experiments of the two insole schemes. In contrast, the data curve of solution No. 1 is "messy", and the mess is mainly reflected in the insufficient regularity of the sensor signal fluctuations located in the toe area.

**Figure 11.** Signal Curve of the second insole solution.

To further compare the quality of the two solutions, the pressure centers are calculated by Equations (1) and (2), as shown in Figures 12 and 13. From the *CoPy* of the curve of solution No. 1, we can observe that the plantar pressure center during the stance phase is disturbed suddenly and, then, the *CoPy* of the curve of the solution No. 2 can better reflect that the pressure center is standing. The tendency of the phase is to move from the heel to the toe, therefore the layout of the second solution is regarded as a better sensor layout.

**Figure 12.** COP values of the first insole solution.

#### 3.2.2. Biped Gait Data Collection

We completed the left foot insole production according to the same production process as the right foot insole. Since the circuit system design reserved up to 16 sensing channels for both feet, one only needs to open the left foot sensing channel in the acquisition program to acquire all the sensing channels of the feet (a total of 12 sensing units). The equipment used for the bipedal gait data collection experiment is shown in Figure 14, including a pair of versatile 43 size shoes and a pair of self-designed two-point photoelectric pressure sensor insoles. Before collecting data, the wearer uses a nylon bayonet to fix the wire on the back of the lower limbs, in the manner shown in Figure 14, to avoid the influence of wire swinging on the usual walking movement during walking. The nylon bayonet and cable tie are fixed on the wearer's waist, and the power bank can be placed in the wearer's trouser pocket after power is supplied to achieve the minimum hindrance to walking. The gait data of a subject was collected using the device. The subject has never suffered from any disease that hinders walking posture. The experiment process is also from a natural standing state to a waking state with an average pace followed by staying still for a few seconds.

**Figure 13.** COP values of the second insole solution.

**Figure 14.** The experimental setup. (**a**) equipment used (**b**) sensing system setup back view (**c**) sensing system setup front view.

(**a**) (**b**) (**c**)

Figures 15 and 16 describes the dynamic bipedal walking experiment data. Observing the data, each sensor presents a periodic "rest state" and "active state". During the "active state", the sensor signals reach their respective peaks in succession. During the "resting state", all sensors returned to their lower levels, which is consistent with our intuitive impression of the phase of standing support and swing phase during the complete gait cycle of a single leg.

**Figure 15.** Right foot plantar pressure sensor signal during the bipedal walking experiment.

**Figure 16.** Left foot plantar pressure sensor signal during the bipedal walking experiment.

In addition, it can be estimated that the overall walking frequency is about 31 steps per minute from the periodically changing curve. Focusing on the data of each sensor of the insole, we can observe that, during the initial period from the "rest state" to the "active state", the No. 1 sensor unit located on the heel first senses the pressure and quickly rises to the peak; then, as the pressure of the No. 1 sensor gradually decreases, the pressure of the No. 2 sensors located on the outside of the arch of the foot also change; secondly, the No. 3, No. 4 and No. 5 sensors located on the forefoot and the toes were in the same interval, reaching their respective peaks over a long time; finally, all sensor signals return to a lower level of resting state. Connecting the pressure curves of the left and right feet, we can see that in the short period when the pressure of the right foot is about to enter a lower level, the heel pressure of the left foot has already been generated and rapidly increased to the peak. Similarly, the pressure of the left foot is about to enter a higher level. During the short period of low level, the heel pressure of the right foot has also begun to reach its peak rapidly, which also reveals the "bipedal standing phase" that is not easily noticeable when walking. Both feet are in contact with the ground during walking.

#### **4. Discussion**

The content shown in this research is mainly focused on the integration of photoelectric sensing technology into the pressure sensing unit to achieve the purpose of sensing plantar pressure information, including the analysis of the photoelectric sensing principle and technology, and the introduction of the technical method realization process. We conducted a test and result analysis of sensor unit characteristics, and performance evaluation during actual use. Here, we will discuss and analyze the results obtained, and make reasonable assumptions and note prospects for further research work.

In this study, in terms of the design of the modular sensing unit, the modular-type sensor provides flexibility for the layout of the pressure sensing insole solution. This modular sensor unit is designed ingeniously and economically. The production of the sensor unit can be completed by using some materials that can be easily purchased from the market. However, as far as the manufacturing method is concerned, the method provided in this article is only for small batches and is a hand-made method, so there are certain defects in the stability and repeatability of the sensor characteristics, which can be found in the sensor characterization section. The result analysis shows that in further research work, if one wants to obtain a more stable and reliable modular sensor unit, it is needed to improve the existing processing technology and manufacturing equipment, and choosing a mature engineering technology may be able to solve this problem. On the other hand, it is worth mentioning that our research introduced a compromised pressure analysis instrument, which can carry out pressure-related calibration test work by building a simple force plate and transforming a desktop-level 3D printer. The design of a programmable calibration instrument derived from the analysis of the mechanical and electrical characteristics of the sensor in this study can also be useful for research teams with limited experimental conditions, that is, who cannot obtain equipment with higher precision and more comprehensive functions.

As for the layout of the pressure sensing insole, in some research work, high-resolution intensive plantar pressure sensing is settled as the research target, such as the research work from [20,23,24]. However, from the perspective of practical application, high resolution means a sharp increase in the number of sensor units. The consequence is that more complex computing processors to deal with the hypermultiplet-channel signal and larger power supply units are required to maintain long-term data recording. However, the plantar pressure distribution is continuous, which means there is information redundancy in the same area. The modular sensor unit can be used to lay out the area with main pressure characteristics (such as the pressure sensing insole layout solutions in this article) to reduce the density of the sensing element. In fact, by adjusting the layout position of the sensing unit, the stability and accuracy of the acquisition of plantar pressure gait data can be improved in a controllable manner as exhibited in this research. On the one hand, the experimental results show that, due to the structural configuration of the sensing unit itself, there exists a phenomenon of "negative pressure" around the sensing unit. We analyzed the cause of this phenomenon and used the rectification activation function to preprocess the negative signal. On the other hand, under normal circumstances, the pressure sensor can better monitor the changes in plantar pressure in the position where the force is more extensive. The sensor unit located in the toe area may not be selected as the pressure sensor unit due to the need to control the stability–sensation area. We mainly focused on the analysis of the results of COP in the front and rear directions, which also showed a specific rule; that is, during the period when the observed leg is in the standing support phase, the movement trend of the center of pressure is to shift from the heel position to the toe position gradually. From this information, the following stages of gait can be preliminarily observed:

However, this article only shows a hand-made/manual operation of modifying the sensor layout. In view of the increasing application of machine learning and other technologies in the engineering field, if the sensor layout can be used as an optimization goal, optimization methods based on machine learning can be employed to perform a much better sensing unit layout for a wider range of plantar pressure distribution information. We believe that the optimization strategy using artificial intelligence algorithms will bring reference significance for better design layout.

What is more, when discussing this research work from the perspective of signal transmission, the current design work adopts the method of transmitting the signal through the cable to obtain the pressure signal. It needs to be admitted that although, in our design, the cables are carefully routed according to the path that does not hinder the movement as much as possible, it is inevitable that there are annoying obstacles such as position displacement and winding during the actual locomotion. Wires will negatively affect the reliability and complexity of wearable devices (usually tied to the human body). Therefore, in further research work, wireless communication protocols (such as WIFI, Bluetooth) are urgently desired for signal transmission. The development of a wireless transmission version of the plantar pressure measurement system can get rid of the annoying winding that hinders movement and improve the integration of the entire sensor measurement system, providing a more convenient interface for further integration into wearable applications.

#### **5. Conclusions**

The design and application of a simple and reliable plantar pressure data acquisition device is very important for wearable human body assist equipment such as exoskeletons and power prosthesis. In this research, we conducted a novel plantar pressure sensing insole based on photoelectric sensing technology. The innovation of this modular sensing unit focuses on sensing principles, structural design and elastic materials. We introduced how to use low-cost manufacturing methods and materials to fabricate this modular sensor so that other researchers can easily replicate and further develop related research work. The designed modular pressure sensor realizes calibration analysis on a self-designed programmable calibration instrument. The electromechanical performance evaluation experiment proves that the modular pressure sensor has special applicability in pressure sensing. Subsequently, based on the analysis of plantar pressure, we proposed and compared different pressure-sensitive insole layout solutions, and integrated the modular pressure sensor into the pressure-sensitive insole. The dynamic locomotion results showed that the pressure-sensing insole based on the photoelectric effect can capture the deformation of the insole caused by plantar pressure during walking, sense the distribution of plantar pressure and provide reliable gait-related parameters with no interference to the wearer. This research provides a reliable gait information acquisition device for wearable applications such as powered exoskeletons, prosthesis and orthotics.

**Author Contributions:** Conceptualization, B.R. and J.L.; methodology, B.R. and J.L.; software, B.R. and J.L.; validation, B.R.; formal analysis, J.L.; investigation, B.R. and J.L.; resources, B.R.; data curation, B.R.; writing—original draft preparation, J.L.; writing—review and editing, B.R; visualization, J.L.; supervision, B.R.; project administration, B.R. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the National Natural Science Foundation of China (Grant No. 51775325); National Key Research and Development Program of China (Grant No. 2018YFB1309200); Young Eastern Scholars Program of Shanghai (Grant No. QD2016033).

**Data Availability Statement:** The datasets generated and/or analysed during the current study are available from the corresponding author on reasonable request.

**Acknowledgments:** The authors would like to thank the anonymous reviewers for their constructive comments that will help us to improve the quality of this manuscript.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

## *Article* **Attention-Based 3D Human Pose Sequence Refinement Network**

**Do-Yeop Kim and Ju-Yong Chang \***

Department of Electronics and Communication Engineering, Kwangwoon University, Seoul 01897, Korea<EMAIL_ADDRESS>
**\*** Correspondence<EMAIL_ADDRESS>Tel.: +82-2-940-5136

**Abstract:** Three-dimensional human mesh reconstruction from a single video has made much progress in recent years due to the advances in deep learning. However, previous methods still often reconstruct temporally noisy pose and mesh sequences given in-the-wild video data. To address this problem, we propose a human pose refinement network (HPR-Net) based on a non-local attention mechanism. The pipeline of the proposed framework consists of a weight-regression module, a weighted-averaging module, and a skinned multi-person linear (SMPL) module. First, the weightregression module creates pose affinity weights from a 3D human pose sequence represented in a unit quaternion form. Next, the weighted-averaging module generates a refined 3D pose sequence by performing temporal weighted averaging using the generated affinity weights. Finally, the refined pose sequence is converted into a human mesh sequence using the SMPL module. HPR-Net is a simple but effective post-processing network that can substantially improve the accuracy and temporal smoothness of 3D human mesh sequences obtained from an input video by existing human mesh reconstruction methods. Our experiments show that the noisy results of the existing methods are consistently improved using the proposed method on various real datasets. Notably, our proposed method reduces the pose and acceleration errors of VIBE, the existing state-of-the-art human mesh reconstruction method, by 1.4% and 66.5%, respectively, on the 3DPW dataset.

**Keywords:** 3D human mesh reconstruction; 3D human pose estimation; deep neural network

#### **1. Introduction**

Three-dimensional human pose estimation is an important and actively studied problem in computer vision. Various methods have been proposed to generate successful pose estimation results on the basis of deep learning. These methods have been used to address the problem of reconstructing 3D human pose from a single RGB image or video obtained from a monocular camera. Recently, methods for estimating dense 3D mesh beyond sparse 3D joints have been proposed on the basis of a statistical shape model for human body. However, reconstructing 3D human poses from RGB images accurately remains a difficult problem.

Recent methods for 3D human mesh reconstruction extract features from input images on the basis of deep learning and directly regress the pose and identity parameters of a statistical shape model, such as a skinned multi-person linear model (SMPL) [1], from the extracted features. However, in the case of an image including occlusion or an unseen pose that is not included in training data, the network has difficulty in estimating the correct pose. Methods of estimating temporally coherent pose sequences from input videos have shown moderate performance [2–4]. However, the above problems still prevent the existing methods to reconstruct the correct pose in some frames and generate noisy human motion. For example, the top row of Figure 1 shows the results obtained by VIBE [3], a state-of-the-art method for reconstructing 3D human mesh from video. In the 3rd frame, VIBE fails to estimate the correct pose, resulting in temporally noisy results. Our study focusses on this problem, namely temporally coherent human pose estimation from input

**Citation:** Kim, D.-Y.; Chang, J.-Y. Attention-Based 3D Human Pose Sequence Refinement Network. *Sensors* **2021**, *21*, 4572. https:// doi.org/10.3390/s21134572

Academic Editors: Tomasz Krzeszowski, Adam Swito ´ ´ nski, Michal Kepski and Carlos Tavares Calafate

Received: 13 June 2021 Accepted: 1 July 2021 Published: 3 July 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

video. Specifically, we propose a *human pose refinement network* (i.e., HPR-Net) that can refine the noisy human pose sequence reconstructed by existing methods. The bottom row of Figure 1 shows the improved results through our HPR-Net.

#### **Frame sequence**

**Figure 1.** This figure shows a 3D human mesh sequence estimated by VIBE (**top row**) and its refined result by our proposed method (**bottom row**). In the 3rd frame, VIBE fails to estimate the correct pose of the target person due to severe occlusion. Our method effectively refines the incorrectly estimated results.

Weighted averaging is a simple but effective method that has been widely used for refinement of signals, including images. The basic idea of this paper to refine the noisy 3D human pose sequence is based on weighted averaging. However, applying weighted averaging to human pose refinement is not trivial due to the following two problems. The first problem is how to determine weights for weighted averaging. To accomplish this task, we learn a module that generates optimal weights on the basis of large-scale data. Specifically, we define a weight as affinity between two 3D poses. We propose a non-local attention-based weight regression module that can consider long-range interactions to compute this affinity. The proposed module is supervised to output weights that can reconstruct a ground-truth pose sequence from a noisy pose sequence estimated by existing human pose estimation methods.

Our human pose refinement method relies on SMPL, where 3D human pose is represented as a set of 3D rotations of joints. However, 3D rotation and 3D pose, including the rotation, cannot be regarded as a vector defined in Euclidean space. Thus, weighted averaging cannot be applied directly to 3D human poses. Performing weighted averaging for 3D rotation requires a complex optimization process [5]. To alleviate this problem, we use Gramkow's study [6], which proves that the mean of unit quaternions is a quadratic approximation of the mean of 3D rotations. Specifically, we first represent the 3D rotation constituting the 3D human pose as a unit quaternion and then perform weighted averaging on the 3D human pose sequence represented as a sequence of unit quaternions. This weighted averaging based on the unit quaternion can be represented as a simple algebraic equation without an optimization process and can be included in our network for learning.

Suppose that SMPL-based human mesh reconstruction methods estimate the 3D pose and identity parameter sequence. Our proposed system consisting of a weight-regression module, a weighted-averaging module, and an SMPL module performs pose refinement for a noisy 3D pose sequence through the following process. To refine a pose of a frame, the weight-regression module first generates weights for the poses in a window of a predefined size around that frame. Next, the weighted-averaging module outputs an improved 3D pose by applying weighted averaging on the basis of the generated weights to the poses inside the window. Finally, the SMPL module generates human meshes and 3D joints from the improved 3D pose parameters. This process is repeated for all frames to reconstruct the refined 3D pose and mesh sequence. An overview of the proposed method is shown in Figure 2.

**Figure 2.** Overall framework of the proposed method. The input to our model is a noisy 3D human pose sequence estimated by existing 3D human pose estimation methods. Our proposed HPR-Net refines the noisy 3D human pose sequence and generates a refined human pose sequence.

The contributions of this paper can be summarized as follows:

#### **2. Related Work**

**Human mesh reconstruction.** Many recent 3D human mesh reconstruction methods directly regress the parameters of statistical shape models, such as SMPL [1]. These methods can be broadly classified into a single image-based approach [7–10] and a video-based approach [2–4].

The single image-based approaches reconstruct 3D human mesh from a monocular image. Bogo et al. [7] proposed a method that estimates 2D joints from an input image on the basis of a pretrained 2D joint regression network and optimizes an energy function to fit SMPL to the regressed 2D joints. Pavlakos et al. [8] extended [7] to optimize an improved energy function to fit the SMPL-X model to the regressed 2D full-body joints for holistic body modeling. A variational autoencoder(VAE)-based pose prior for valid pose parameter regression was proposed for optimization. Kanazawa et al. [9] proposed a model that directly maps features extracted by a deep network from a single image to SMPL parameters. In their method, an adversarial prior for the estimated parameters was proposed and learned to help obtain a realistic human mesh. Kolotouros et al. [10] combined the optimization-based method and the regression-based method in an endto-end manner. The SMPL parameter estimated from a single image is used as an initial parameter, which is iteratively optimized through the method of [7]. The optimized

parameter is used as a pseudo ground-truth for learning the regressor to construct a self-improving framework.

The video-based approaches reconstruct the 3D human mesh sequence from a video. Kanazawa et al. [2] proposed a temporal convolutional network that reconstructs the SMPL model from an image sequence. This method is supervised to predict SMPL models in the nearby few frames to learn information about human motion better. It can estimate past and future meshes from a single image through a hallucinator. Kocabas et al. [3] proposed a method that reconstructs an SMPL model sequence from a feature sequence computed using bidirectional gated recurrent unit from an input video. To compensate for the lack of 3D annotated data, this method performs weak supervision with various 2D datasets and adversarial training using large-scale motion datasets, resulting in successful human mesh reconstruction performance. Luo et al. [4] tried to solve the jittering problem from the inference results of existing methods for video data. This method reconstructs coarse motion by learning a VAE-based motion prior and then performs refinement for each frame's pose. Thus, the smoothness of the output SMPL sequence is improved. Despite these recent advances in 3D human mesh reconstruction, most methods still produce erroneous poses or jittered motions due to unseen poses or occlusions from input images or videos acquired in an uncontrolled environment. Our work can substantially improve the accuracy and smoothness of human mesh sequences reconstructed by existing methods.

**Non-local attention.** Non-local attention was proposed to model long-range dependency in natural language processing [11,12] and computer vision [13–15]. Vaswani et al. [12] proposed the transformer, which is a framework using only attention mechanisms to overcome the limitations of existing recurrent models for natural language processing tasks and successfully solves the long-range dependency problem. Recently, the transformer architecture is known to improve image recognition performance and is actively used for various computer vision tasks [16–20]. Wang et al. [13] attempted to model the long-range dependency in image features using non-local operations proposed in [21]. For this, a non-local block based on attention mechanisms was proposed. On this basis, the method in [12] can be regarded as a special case of non-local neural networks. In the study of Cao et al. [14], the position-wise attention map of [13] was analyzed qualitatively, and most of the attention maps of each position have similar attention aspects. On this Basis, a more efficient non-local attention block was proposed. Woo et al. [15] proposed a method that extracts new features by successively applying channel attention and spatial attention to input features. This method shows a stronger representation power than features based on existing fully convolutional baselines. Our method generates a temporal non-local attention map inspired by [13,21]. The generated attention weights suppress features that are useless for refinement and strengthen helpful features. Our method can refine noisy pose parameter sequences through this attention mechanism.

**Human pose refinement.** The goal of human pose refinement studies is mainly to refine an estimated sparse joint set. Existing pose refinement methods are included as part of the joint regression network or used as a post-processing module for inference results. Newell et al. [22] proposed a network in which several hourglass modules are stacked. Hourglass module repeats top-down and bottom-up processing, extracts features at various scales, and is trained with intermediate supervision. Each stage module generates a heatmap, which is used as input to the next stage module for refinement. Chen et al. [23] proposed a cascaded pyramid network that combines GlobalNet, a Resnet-based pyramid network, and RefineNet, which refines the heatmap generated by GlobalNet. RefineNet considers all features obtained from each step of the pyramid to find occluded joint positions that are difficult to estimate. Moon et al. [24] proposed a model-agnostic refinement model based on the error distribution of 2D pose estimation models investigated in Ronchi et al.'s work [25]. This method is independent on the pose estimation model because it does not work in an end-to-end manner, and pose estimation performance can be improved for various existing approaches. Mall et al. [26] proposed a method to refine noisy motion capture data. The proposed network consisting of linear layers and bidirectional long

short-term memories regresses the standard deviation of a Gaussian kernel to improve the pose of a current frame. The proposed method obtains a denoised pose using the Gaussian kernel obtained through this network to calculate a temporally weighted sum for an input noisy pose sequence. In [26], 3D human pose is represented in the form of 126 joint angles, and the weighted sum is computed for this joint angle sequence. Our work provides a more reliable basis for computation in non-Euclidean space where 3D rotation actually exists. While the values of weights are limited by the Gaussian kernel in [26], they are not in our method.

Several methods have been proposed to refine the SMPL pose parameter [9,10]. Kanazawa et al. [9] proposed a regressor that performs iterative refinement to estimate the SMPL parameter. Kolotouros et al. [10] presented a method that refines the estimated SMPL parameter through an optimization process. In [9,10], the refinement process is included in the model, which outputs SMPL identity and pose parameters directly from an input image. Our refinement method is independent of the pose estimation model and can be applied to the results of any method for estimating the SMPL pose parameter sequence regardless of their network structure. Our work is the first to propose a post-processing method for SMPL pose parameter refinement, and the proposed method is simple but works effectively.

#### **3. Proposed Method**

This section provides detailed descriptions of each module constituting our proposed HPR-Net. As presented in Section 1, we propose HPR-Net that generates a refined 3D human pose sequence from a noisy 3D human pose sequence estimated by other methods, such as VIBE. As shown in Figure 2, HPR-Net refines a noisy 3D pose of a target frame from input 3D poses that consist of all 3D poses within a window of size *N* (*N* is an odd number) centered on the target frame. We term these input 3D poses as a *pose chunk*. HPR-Net consists of a weight-regression module, a weighted-averaging module, and an SMPL module. Each module is explained in the following subsections. We first introduce the SMPL module to explain what 3D human model is used, how the 3D human pose is defined in the SMPL model, and why this module is needed in our framework. The weightregression module consists of 1D convolution layers and generates an *N*-dimensional weight vector using non-local self-attention mechanism from an input pose chunk. The weighted-averaging module outputs a refined 3D pose by weighted averaging with the input pose chunk and weights from the weight-regression module. The above procedure is applied to the noisy 3D pose sequence with a sliding window manner, so we can obtain the refined 3D human pose sequence.

#### *3.1. SMPL Module*

SMPL is a 3D statistical shape model used to represent a human body and includes low-dimensional parameters to control the body shape. The parameter set included in the SMPL model consists of an identity parameter *β* ∈ R10 and a pose parameter *θ* ∈ R72. The pose parameter represents the relative 3D rotations of 24 joints in an axis-angle form. This parameter controls the 3D pose of the human body represented by the SMPL model. From the given identity and pose parameters, the SMPL module generates a 3D human mesh model in a differentiable manner. The vertices *M* ∈ R3×6890 of the generated mesh model are multiplied with a pretrained linear regression matrix included in the SMPL model, so that 24 joints *Xsmpl*,3*d* ∈ R3×24 can be additionally obtained. In HPR-Net, the SMPL module computes the human mesh model and the 3D joints from the refined pose parameters using our weighted-averaging module and the identity parameters estimated by existing methods. We can compare the 3D joints from the refined mesh generated by the SMPL module with its ground-truth to compute a loss function for learning and error for evaluating the proposed method. We also use the joint set *X*3*d* ∈ R3×14 obtained by converting *Xsmpl*,3*d* into 14 joints compatible with the joint definition of Human3.6M [27] for learning and evaluation.

#### *3.2. Weight-Regression Module*

**Network structure.** The weight-regression module of HPR-Net generates weights for pose refinement of the target pose from an input noisy pose chunk. Figure 3 shows the detailed structure of the weight-regression module consisting of 1D temporal convolution layer with a kernel size of 3, layer normalization [28], rectified linear unit activation, and self-attention layer. Suppose that Φ = {*βi*, *θi*}*N*−1 *i*=0 , which is a chunk of length *N* for the noisy SMPL parameter sequence, is given. Here, *βi* ∈ R10 and *θi* ∈ R72 are the identity and pose parameters in the *i*-th frame, respectively. The pose parameter *θi* represents the 3D rotations for 24 SMPL joints represented in an axis-angle form. We first convert *θi* to pose parameter *pi* ∈ R96 in a unit quaternion form. We apply frame-wise positional encoding to the unit quaternion pose chunk *P* = [*p*0, ... , *pN*−1] ∈ R96×*N*, similar to [12], before feeding it into the network. Specifically, to inject positional information into *P*, we concatenate a relative position index vector [− *N* 2 , ... , −1, 0, 1, ... ,  *N* 2 ] with *P* to construct *P*˜ ∈ R97×*N* and feed the concatenated tensor into the weight-regression module. The weight-regression module first computes temporal feature *H* = [*h*0, *h*1, ... , *hN*−1] ∈ R24×*N* from *P*˜ through three 1D temporal convolution layers. *hi* ∈ R24 represents the temporal feature of the *i*-th frame. A pose affinity vector *w* ∈ R*N* is generated through a non-local self-attention mechanism [12,13] as follows:

$$\varpi = \text{Softmax}(H^T \cdot \mathbb{H}\_{\lfloor \frac{N}{2} \rfloor}).\tag{1}$$

Similar to the existing self-attention-based methods, our weight-regression module simply uses matrix multiplication and softmax operation to construct pose affinity vector *w* = [*w*0, *w*1, ... , *wN*−1] *T*, where *wi* ∈ R represents how the pose of *i*-th frame affects the computation of the refined pose at the  *N* 2 -th (i.e., center) frame. As our HPR-Net refines the center frame's pose *p N* 2 from *P*, we choose the center frame's feature *h N* 2 from *H* to compare the feature with all other features within the chunk.

**Figure 3.** Detailed pipeline of the weight-regression module. ⊗ represents matrix multiplication. First, the weight-regression module concatenates positional information to an input pose chunk. Second, the positional encoded input chunk is fed into the weight-regression module that consists of three 1D temporal convolution layers. Finally, pose affinity vector is generated from the output temporal feature of the convolution layers.

**Why do we use LayerNorm?** From our experiments, we observed that the use of layer normalization after the convolution layer shows higher performance than the commonly used batch normalization [29]. In our method, the 3D pose in an input pose chunk consists of 3D rotations, and this 3D rotation is represented in a unit quaternion form that is geometrically on a 4D unit sphere. Layer normalization helps to learn the weight-regression module by enforcing the features extracted through the convolution layer to be on the unit sphere.

#### *3.3. Weighted-Averaging Module*

**Pose refinement by weighted averaging.** Using *w* generated by the weight-regression module, we perform weighted averaging on the input pose chunk *P* and obtain the refined pose *y* ∈ R96 as follows. Figure 4 shows the detailed structure of the weighted-averaging module. Weighted averaging cannot be directly applied to 3D rotations because they are defined in non-Euclidean space. Therefore, we obtain a second-order approximation of optimal rotation averaging by performing weighted averaging based on unit quaternion following Gramkow's work [6]. By weighted averaging, we first obtain ˜*y* as follows:

$$
\bar{y} = \sum\_{i=0}^{N-1} w\_i p\_{i\prime} \tag{2}
$$

where *wi* is the *i*-th component of vector *w* and represents the contribution of *pi* to weighted averaging. However, *y*˜ = [*q*˜*T* 0 , *q*˜*T* 1 , ... , *q*˜*T* 23] *T* cannot be guaranteed to consist of unit quaternions. Therefore, we additionally perform normalization to make the 3D rotations *q*˜*j* belonging to *y*˜ into a unit quaternion form using *qj* = *q*˜*j*/ *q*˜*j* . The weighted-averaging module outputs the refined 3D pose *y* consisting of unit quaternions as follows:

$$y = [q\_0^T, q\_1^T, \dots, q\_{23}^T]^T \in \mathbb{R}^{96},\tag{3}$$

where *qj* denotes the 3D rotation of the *j*-th SMPL joint.

**Figure 4.** Detailed pipeline of the weighted-averaging module. represents element-wise multiplication with broadcasting. Σ represents summation for across time dimension. Input pose vectors *P* are multiplied with pose affinity weights *w* which are generated by the weight-regression module. Then weighted pose vectors are added to output a refined pose vector *y*˜. To ensure that the refined pose parameters consist of unit quaternions, we additionally normalize *y*˜ to output a valid pose vector *y*.

**Loss functions.** The refined 3D human pose *y* is converted to an axis-angle form and then fed into the SMPL module along with the identity parameter *β* estimated by other methods to generate the refined mesh *M*ˆ and 3D joints *X*ˆ 3*d* = [*x*ˆ3*d*,1, ... , *x*ˆ3*d*,14]. The joint loss function *L*joint for learning the proposed network is defined as follows:

$$L\_{\text{jcint}} = \frac{1}{J} \sum\_{j=1}^{J} ||\hat{\mathbf{x}}\_{3d,j} - \mathbf{x}\_{3d,j}||\_{1, \epsilon} \tag{4}$$

where *J* = 14 is the number of joints, and *x*ˆ3*d*,*j* and *x*3*d*,*j* denote the estimated and groundtruth coordinate vectors of the *j*-th joint, respectively. *L*joint is defined as L1 loss and we supervise only *X*ˆ 3*d* that is generated from *M*ˆ by the SMPL module.

#### **4. Experimental Results**

#### *4.1. Datasets and Evaluation Metrics*

We use Human3.6M [27] and 3DPW [30] for training and evaluation. Human3.6M is a large-scale dataset obtained from an indoor environment, and has been used in many existing 3D human pose estimation methods. The Human3.6M dataset consists of videos, where 11 subjects perform 15 actions and includes 2D and 3D joint annotations for each

frame. Image data in Human3.6M were captured in four camera views. 3DPW is a dataset obtained in an in-the-wild environment. The 3DPW dataset includes 60 videos and is divided into train, validation, and test sets. The three sets consist of 24, 12, and 24 videos, respectively. The 3DPW dataset provides 2D and 3D joint annotations and SMPL parameter annotations.

We split each dataset into training and test data. In Human3.6M, we use 5 subjects (1, 5, 6, 7, 8) as training data and 2 subjects (9, 11) as test data following the convention of previous studies [3,4,10]. For the 3DPW dataset, we use the train and validation sets as training data and the test set as test data. For convenience of training and evaluation, we apply VIBE to the training data of Human3.6M and 3DPW datasets and store the estimated SMPL parameters offline. The saved results are used as input for training the proposed network. We apply SPIN [10], VIBE [3], and MEVA [4] to the test data of each dataset, store the estimated SMPL parameters offline, and use them as input for evaluation. At the training stage, we train the proposed HPR-Net using all the training data of each dataset. We then evaluate the proposed method by applying HPR-Net to the test data of each dataset and report the performance quantitatively and qualitatively.

To evaluate the performance of the proposed method, we report MPJPE, PA-MPJPE, MPVE, and acceleration error. MPJPE and PA-MPJPE are metrics used to evaluate joint position error. MPJPE calculates the average 3D joint distance (mm). PA-MPJPE calculates the average 3D joint distance (mm) after performing Procrustes alignment [31] on the estimated and ground-truth joint sets. MPVE calculates the average position error (mm) of the vertices of the generated SMPL mesh. Acceleration error [2] is a metric for evaluating the temporal smoothness (mm/s2) of the estimated pose sequence. Acceleration vectors are computed for the 3D joint sequence, and the acceleration error is calculated as the average difference between the estimated and ground-truth acceleration vectors.

#### *4.2. Implementation Details*

HPR-Net is trained end-to-end, and the input pose chunk in the training process is determined by random shuffling at each iteration. Zero padding is applied to the temporal 1D convolution of the weight-regression module. We set the length *N* of the input pose chunk for training the HPR-Net to 17 and calculate the loss function for the joints of the refined mesh corresponding to the center frame. We use Adam [32] as the optimizer of the network. The learning rate is set to 10−4. We do not decay the learning rate during training. The batch size and the number of epochs are set to 64 and 20, respectively. In each epoch, 1000 iterations are performed. The learning rate, batch size, and number of epochs are determined through simple greedy search using the validation set of 3DPW. Pytorch [33] was used to implement the proposed method, which was trained with a single Nvidia RTX3090 GPU. In the evaluation process, the input pose chunk is not randomly determined and is fed into the network in the order of the frames of the evaluation video. We refine the input video except for 16 frames (i.e., 8 frames each at the beginning and end of the video). The pose chunk of length 17 is fed into HPR-Net in a sliding window manner with stride 1.

#### *4.3. Ablation Study*

In ablation experiments, we report how the hyperparameters and component changes affect the performance of HPR-Net. We use VIBE's pose sequence estimation result as our HPR-Net's input chunk. We set the length of the input pose chunk to 17 in all experiments, except for the pose chunk length ablation experiment. In ablation experiments, HPR-Net is evaluated on 3DPW test set.

**Pose chunk length.** To determine the optimal length *N* of the pose chunk, we perform training using various lengths and analyze the results. Table 1 shows the performance in accordance with the length of the input chunk. HPR-Net shows the best performance with length 17, except for PA-MPJPE. Thus, we set the pose chunk length to 17.

**Table 1.** Performance comparison of HPR-Net according to different pose chunk length. Bold values indicate best results.

**Various loss combinations.** In the proposed method, only 3D joints are supervised to train HPR-Net using the joint loss function in Equation (4). To justify this condition, we conduct an experiment to investigate how various combinations of loss functions affect the performance of HPR-Net. Specifically, we perform direct supervision with the joint loss function *L*joint and losses that can be defined using the outputs of HPR-Net. The mesh loss function *L*mesh and the pose loss function *L*pose are additionally defined as follows:

$$L\_{\text{mech}} = \frac{1}{6890} \sum\_{\upsilon=1}^{6890} ||\dot{m}\_{\upsilon} - m\_{\upsilon}||\_{1\prime} \tag{5}$$

$$L\_{\text{pose}} = \frac{1}{24} \sum\_{j=1}^{24} \|\mathcal{R}\_j - \mathcal{R}\_j\|\_F^2. \tag{6}$$

The mesh loss function *L*mesh is defined as L1 loss, where *m*ˆ *v* and *mv* denote the estimated and ground-truth coordinate vectors for the *v*-th vertex, respectively. The pose loss function *L*pose is for the pose parameters, including 3D rotations, where *R*ˆ *j* and *Rj* ∈ R3×3 denote the estimated and ground-truth rotation matrices for the *j*-th joint, respectively. Frobenius norm for their difference represents the distance (i.e., chordal distance [5]) between two 3D rotations in non-Euclidean space. The total loss function *L* for this ablation experiment is defined as follows:

$$L = \lambda\_{\text{j}} L\_{\text{joint}} + \lambda\_{\text{m}} L\_{\text{mresh}} + \lambda\_{\text{p}} L\_{\text{pcosa}} \tag{7}$$

where *λj*, *λm*, and *λp* denote the weights that determine the strength of each loss.

Table 2 shows the performance of HPR-Net in accordance with the weights of *L*. HPR-Net shows the highest performance, except for PA-MPJPE when only the joint loss function *L*joint is used. Using the pose loss function *L*pose leads to performance degradation (1st, 4th, 6th, 7th rows). Supervising the mesh vertices shows lower PA-MPJPE (2nd row) than only using *L*joint (3rd row). We use Human3.6M and 3DPW datasets for training. However, the Human3.6M dataset does not include SMPL annotations. Thus, only the 3DPW dataset is used to supervise the network when we calculate *L*mesh and *L*pose. The size of 3DPW training data is smaller than that of Human3.6M. However, the experimental result from supervising with only *L*mesh shows the highest PA-MPJPE and second highest performance on other metrics. If more datasets containing SMPL annotations are available, then the use of the mesh loss function will lead to further performance improvements.

**Table 2.** Performance comparison of HPR-Net according to various combinations of loss functions. (= 1.0, blank = 0.0). Bold values indicate best results.

**Positional encoding.** Most of the non-local attention-based methods inject positional information into their input. HPR-Net performs positional encoding, which helps to distinguish the pose of each frame in input pose chunk. We investigate the effect of positional encoding and its method on the performance of HPR-Net. Table 3 shows the performance of HPR-Net in accordance with the positional encoding method. For the experiment, we train and evaluate with three different models, one without positional encoding (*None*), one with sinusoidal positional encoding according to [12] (*Sinusoidal*), and one with positional encoding used in the proposed method (*Ours*). When positional encoding is not used, HPR-Net shows decreased PA-MPJPE performance compared with VIBE, but the other metrics are improved. Using the sinusoidal positional encoding shows improved results and best performance on PA-MPJPE. Our encoding method shows slightly lower PA-MPJPE compared with the sinusoidal positional encoding, but the best performance on the other metrics.

**Table 3.** Comparison of refinement performance of HPR-Net according to positional encoding method. Bold values indicate best results.

**Layer normalization.** The weight-regression module is composed of simple 1D temporal convolution layers. Layer normalization is adopted as the feature normalization layer of the proposed weight-regression module. To justify the use of layer normalization for HPR-Net, we trained three models, one without feature normalization, one using batch normalization, and one using layer normalization. Table 4 shows the performance comparison in accordance with the normalization method used in HPR-Net. When layer normalization is used, HPR-Net achieves the best performance in all metrics compared with other methods. From the result, layer normalization helps the learning of the weightregression module.

**Table 4.** Comparison of refinement performance of HPR-Net according to feature normalization method. Bold values indicate best results.

#### *4.4. Refinement on State-of-the-Art Methods*

We evaluate the performance of applying HPR-Net to state-of-the-art methods [3,4,10] for different datasets [27,30]. Tables 5 and 6 report the performance of existing methods and their refinement performance by HPR-Net on each evaluation dataset. Existing methods are re-evaluated using publicly provided pretrained models. HPR-Net achieves performance improvement in all metrics for all methods on 3DPW and Human3.6M datasets. HPR-Net considerably improves the acceleration error in every experiments. We trained our HPR-Net with the pose estimation result by VIBE as input. However, HPR-Net consistently improves other methods (i.e., SPIN and MEVA). These results show our HPR-Net's generalization capability for other methods.

**Table 5.** HPR-Net's pose refinement performance for various existing methods on 3DPW test data. Bold values indicate performance improvements.

**Table 6.** HPR-Net's pose refinement performance for various existing methods on Human3.6M test data. Bold values indicate performance improvements.

#### *4.5. Comparison with Other Pose Refinement Methods*

The pose parameter sequence can be refined in several methods. We compare HPR-Net with other methods in improving the pose sequence. Table 7 shows the quantitative improvement results of SLERP, Gaussian-filtering-based method (HPR-Gaussian), directregression-based method (HPR-DR), and HPR-Net. All the methods are evaluated on 3DPW test set. SLERP calculates the interpolated unit quaternion between two unit quaternions. MEVA uses SLERP to further smoothen their output pose parameter sequence. We test SLERP to evaluate its refinement performance and compare it with our HPR-Net's performance. HPR-Gaussian regresses standard deviations to create optimal joint-wise Gaussian kernels. We implement the HPR-Gaussian model by modifying the structure of the weight-regression module in HPR-Net. We only change the kernel size of the 3rd temporal 1D convolution layer of the weight-regression module to *N* and set the number of channels to 24. HPR-Gaussian's weight-regression module creates 24 joint-wise standard deviations, where the 24 Gaussian kernels with kernel size *N* are created. Each kernel is used for Gaussian filtering for the 3D rotation of each of the 24 joints. Specifically, weighted averaging of 3D rotations along the temporal axis is performed using the values of the kernels as weights. HPR-DR directly regresses the refined pose of the center frame from the input pose chunk. To implement HPR-DR, our proposed HPR-Net is modified as follows. We change the number of channels and kernel size in the last 1D convolution layer of the weight-regression module to 96 and *N*, respectively, so that the modified network (i.e., HPR-DR) generates a 96D vector. This vector is converted into a refined pose vector consisting of unit quaternions through normalization.

**Table 7.** Comparison of refinement performance between HPR-Net and other pose sequence refinement methods on the 3DPW dataset. Bold values indicate best results.

From the quantitative improvement results of each method in Table 7, we observe that SLERP does not improve the performance, except for the acceleration error. Acceleration is defined as the second derivative of the joint position and is very sensitive to the small noise in the refined pose sequence. Since SLERP performs weighted averaging for interpolation between two poses, it is effective in reducing the small noise and the acceleration error. HPR-Gaussian improves VIBE quantitatively. However, the performance gain for the acceleration error is smaller than SLERP because HPR-Gaussian over-smooths the pose sequence. HPR-DR fails to refine the results of VIBE. It is because the size of the training data, which is not large enough to train the direct regression model, leads to overfitting. Our HPR-Net adaptively adjusts the shape of the kernel to prevent over-smoothing and outperforms the other methods in all metrics, especially the acceleration error. Experimental results show that our HPR-Net is superior to the other human pose refinement methods.

#### *4.6. Network Design Based on Non-Local Attention*

Our proposed HPR-Net is based on non-local attention. Transformer [12] is a representative method and has the non-local attention-based structure. Our HPR-Net's network structure is similar to that of the Transformer's non-local self-attention module, but HPR-Net does not include components, such as multi-head attention and linear projection. To explore how these components affect our model, we compare our HPR-Net with two HPR-Net variants with a multi-head attention structure (MHA) and a single-head attention structure (SHA). The details of each structure are shown in Figure 5.

**Figure 5.** Detailed pipelines of multi-head structure (**a**), linear projection structure (**b**), and our proposed HPR-Net's structure (**c**) for network design experiment. We did not apply linear projection to input pose chunk *P* in (**a**–**c**), because it should be averaged with affinity weights. Attention head contains affinity vector generation by self-attention and weighted-averaging processes.

Table 8 shows the VIBE refinement performance of MHA, SHA, and our HPR-Net on the 3DPW dataset. All the experimented structures show acceleration error improvement. MHA shows higher MPJPE, PA-MPJPE, and MPVE than SHA. However, the two models fail to improve MPJPE, PA-MPJPE, and MPVE compared with VIBE. Unlike the two structures, HPR-Net improves the performance in all metrics and achieves the lowest acceleration error. The difference between our method and the two structures are that the features obtained from the convolution layers are not linearly projected, and the MHA is not used in the proposed HPR-Net. From the results, the linear projection layer seems to cause performance degradation by confusing to generate an appropriate affinity weight vector from input pose information. The MHA seems to result in overfitting by complicating the network structure more than necessary. Our network has a simpler structure and performs

better. HPR-Net is more optimal in solving our problem than the commonly used non-local attention structure.

**Table 8.** Comparison of refinement performance according to network design of HPR-Net. Bold values indicate best results.

#### *4.7. Qualitative Results*

**Acceleration error improvement.** HPR-Net consistently shows a significant improvement in acceleration error across all methods and datasets on the basis of quantitative results. We present qualitative improvement results using a graph. Figure 6 shows the acceleration error of VIBE, SPIN, MEVA and their refined results after applying HPR-Net to each method. The acceleration errors are calculated for every three consecutive frames from a video of 3DPW. Compared with existing methods' result, HPR-Net effectively improves the acceleration errors for all methods. In particular, the acceleration error is significantly reduced in frames with high peaks where the errors are noticeable.

**Refinement result.** We present the qualitative results to show that HPR-Net substantially refines a 3D human pose sequence estimated by existing methods. Figures 7 and 8 show the refined results for VIBE and SPIN, respectively. For each example in Figures 7 and 8, the top, middle, and bottom rows show the input image sequence, the estimation result by the existing method, and the refinement result by the proposed HPR-Net, respectively. We do not report the qualitative result for MEVA, because the SMPL estimation results by MEVA's official code are projected incorrectly in the image. In the topmost example of Figure 7, a pedestrian causes occlusion. Thus, the pose of the target subject is incorrectly estimated. HPR-Net refines the results by reconstructing the appropriate pose using the information of nearby frames. In the top-left example of Figure 8, SPIN predicts the global orientation incorrectly due to challenging illumination. This incorrect global orientation is well refined in the result of HPR-Net. From the other results, HPR-Net refines the incorrect estimations of arms and legs.

#### *4.8. Discussion*

In accordance with our experimental results, the refinement of the human pose sequence estimated by existing methods can be achieved through a data-driven approach on the basis of a large-scale dataset and a deep neural network. To realize this, the proposed HPR-Net adaptively performs weighted averaging, a well-known framework for noise reduction, on input data, therefore consistently improving the human pose estimation performance of existing state-of-the-art methods. The pose refinement by HPR-Net is performed independently of the existing human pose estimation method. This modularity can be a benefit of our approach because it makes the use and analysis of the proposed method easy. However, HPR-Net has a limitation of depending on the pose estimation results of existing methods. Combining HPR-Net with the existing pose estimation network and learning it in an end-to-end manner may bring additional performance improvement. We plan to continue our research to investigate the end-to-end approach and overcome the limitations of the proposed method.

**Figure 6.** Comparison of acceleration error between HPR-Net and previous methods (VIBE, SPIN, and MEVA). HPR-Net effectively suppresses acceleration error for all methods, even there are very high peaks of acceleration error.

**Figure 7.** Input images (**top**) and reconstruction results of VIBE (**middle**, gray SMPL mesh) and HPR-Net (**bottom**, yellow SMPL mesh) on the 3DPW dataset.

**Figure 8.** Input images (**top**) and reconstruction results of SPIN (**middle**, gray SMPL mesh) and HPR-Net (**bottom**, yellow SMPL mesh) on the 3DPW dataset.

#### **5. Conclusions**

We propose HPR-Net to refine the noisy 3D human pose parameter sequence. HPR-Net improves the accuracy and temporal smoothness of the 3D human pose sequence through a simple non-local attention-based weighted averaging for a noisy pose parameter chunk represented in a unit-quaternion form. We report quantitatively and qualitatively that the proposed method can improve 3D human reconstruction performance for various real datasets, such as Human3.6M and 3DPW. From the experiments for improving the results of various existing methods such as SPIN, VIBE, and MEVA, a consistent performance improvement is observed regardless of the method used to estimate the input human pose sequence. This finding shows that our method works in a model-agnostic manner. The superiority of HPR-Net is confirmed by comparing it with other approaches that can refine 3D human pose parameters.

**Author Contributions:** Conceptualization, D.-Y.K. and J.-Y.C.; methodology, D.-Y.K.; software, D.- Y.K.; validation, D.-Y.K.; formal analysis, D.-Y.K.; investigation, D.-Y.K.; resources, J.-Y.C.; data curation, D.-Y.K.; writing—original draft preparation, D.-Y.K.; writing—review and editing, J.-Y.C.; visualization, D.-Y.K.; supervision, J.-Y.C.; project administration, J.-Y.C.; funding acquisition, J.-Y.C. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was supported by Samsung Research Funding Center of Samsung Electronics (No. SRFC-IT1901-06) and by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No. 2019R1C1C1008462). The present research has been conducted by the Research Grant of Kwangwoon University in 2021.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**

## *Article* **Gap Reconstruction in Optical Motion Capture Sequences Using Neural Networks**

**Przemysław Skurowski 1,\* and Magdalena Pawlyta 1,2**

**Abstract:** Optical motion capture is a mature contemporary technique for the acquisition of motion data; alas, it is non-error-free. Due to technical limitations and occlusions of markers, gaps might occur in such recordings. The article reviews various neural network architectures applied to the gapfilling problem in motion capture sequences within the FBM framework providing a representation of body kinematic structure. The results are compared with interpolation and matrix completion methods. We found out that, for longer sequences, simple linear feedforward neural networks can outperform the other, sophisticated architectures, but these outcomes might be affected by the small amount of data availabe for training. We were also able to identify that the acceleration and monotonicity of input sequence are the parameters that have a notable impact on the obtained results.

**Keywords:** motion capture; neural networks; reconstruction; gap filling; FFNN; LSTM; BILSTM; GRU

#### **1. Introduction**

Motion capture (mocap) [1,2], in recent years, has become a mature technology that has an important role in many application areas. Its main application is in computer graphics, where it is applied in gaming and movie FX for the generation of realistic-looking character animation. Other prominent applications areas are biomechanics [3], sports [4], medical sciences (involving biomechanical [5] and the other branches, i.e., neurology [6]), and rehabilitation [7].

Optical motion capture (OMC) relies on the visual tracking and triangulation of active or retro-reflective passive markers. Assuming a rigid body model, successive positions of markers (trajectories) are used in further stages of processing to drive an associated skeleton, which is used as a key model for the animation of human-like or animal characters.

OMC is commonly considered the most reliable mocap technology; it is sometimes called the 'gold standard', as it outperforms the other mocap technologies. However, the process of acquiring marker locations is not error-free. Noise, which is immanent in any measurement system, has been studied in numerous works [8,9], which suggests it is not just simple additive Gaussian process. The noise types present in OMC systems were identified in [10]; these are red, pink, white, blue-violet, and Markov–Gaussian-correlated noises; however, they are not a big issue for the mocap operators since they have rather low amplitudes and can be quite efficiently filtered out. The most annoying errors come from marker observation issues. They occur due to marker occlusion and the marker leaving the scene, and result in a lack of the recorded data-gaps that are typically represented as not a number (NaN) values.

The presence of gaps is common and results in everyday praxis, which requires painstaking visual trajectory examination and manual trajectory editing by operators. This can be assisted by software support for trajectory reconstruction.

**Citation:** Skurowski, P.; Pawlyta, M. Gap Reconstruction in Optical Motion Capture Sequences Using Neural Networks. *Sensors* **2021**, *21*, 6115. https://doi.org/10.3390/s21186115

Academic Editors: Tomasz Krzeszowski, Adam Swito ´ ´ nski, Michal Kepski and Carlos Tavares Calafate

Received: 30 July 2021 Accepted: 8 September 2021 Published: 12 September 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

In this work, we propose a marker-wise approach that addresses the trajectory reconstruction problem. We analyze the usability of various neural network architectures applied to regressive tasks. The regression/prediction exploits inter-marker correlations between markers placed on the same body parts. Therefore, we employed a functional body mesh structure (FBM) [11] as a framework to model the kinematic structure of the subject. I Thisan be calculated ad-hoc for any articulated subject or rigid objects, so we do not need a skeleton model.

The article is organized as follows: in Section 2, we disclose the background for the article—mocap pipeline with sources of distortion and former works on the distortions in optical mocap systems; Section 3 describes the proposed method, with its rationales and design considerations, and experiment plan. In the Section 4 we provide results, and a discussion and interpretation of results. Section 5 summarizes the article.

#### **2. Background**

#### *2.1. Optical Motion Capture Pipeline*

Optical motion capture systems track the markers—usually passive retro-reflective spheres in near-infrared images (NIR) images. The basic pipeline is shown in Figure 1. The markers are observed by several geometrically calibrated NIR cameras. The visual wavelengths cut-off, and, hence, the images, contain just white dots, which are matched between the views and triangulated, so the outcome of the early stage of mocap is a time series containing Cartesian coordinates of all markers. An actor and/or object wears a sufficient number of markers to represent body segments—marker layout usually follows a predefined layout standard. The body segments are represented by a predefined mesh, which identifies the body segments and is a marker-wise representation of body structure. Finally, mocap recording takes the form of a skeleton angle time series, which represents the mocap sequence as orientations (angles) in joints and a single Cartesian coordinate for body root (pelvis usually).

**Figure 1.** Stages of the motion capture pipeline: actor (**a**); registered markers (**b**); body mesh (**c**); mesh matched skeleton (**d**).

#### *2.2. Functional Body Mesh*

Functional body mesh (FBM) is a authors' original contribution, that forms a framework for marker-wise mocap data processing, which incorporates also the kinematic structure of a represented object. The FBM structure is not given in advance, but it can be inferred based on the articulated object representative motions [11]. For human actors it resembles standard meshes, but it can be applied for virtually any vertebrates. It assumes the body is divided into rigid segments (submeshes), which are organized into a tree structure. The model represents the hierarchy of subjects' kinematic structure, reflecting

bonds between body segments, where every segment is a local rigid body model—usually based on an underlying bone.

The rigid segments maintain the distance between the markers and, additionally, for each child segment, one representative marker is assumed within the parent one, which is also assumed to maintain a constant distance from the child markers. The typical FBM for the human actor is shown in Figure 2b as a tree. The segments and constituent markers are located in nodes, whereas the parent marker is denoted on the parent–child edge.

**Figure 2.** Outline of the body model (**a**), and corresponding parts hierarchy annotated with parents and siblings (**b**).

#### *2.3. Previous Works*

Gap filling is a classical problem frequently addressed in research on mocap technologies. It was in numerous works, which proposed various approaches. The existing methods can be divided into three main groups—skeleton-based, marker-wise, and coordinate-based.

A classical skeleton-based method was proposed by Herda et al. [12], they estimate skeleton motion and regenerate markers on the body envelope. Aristidou and Lanesby [13] proposed the other method based on a similar concept, where the skeleton is a source for constraints in inverse kinematics estimation of marker location. Also, Perepichka et al. [14] combined IK of skeleton model with deep NN to detect erroneously located markers and to place them on a probable trajectory. All aforementioned approaches require either to have a predefined skeleton or to infer the skeleton as the entry step of an algorithm.

The skeleton-free methods consider information from markers only, usually acknowledging the whole sequence as a single multivariable (matrix), thus losing the kinematic structure of the represented actor. They rely on various concepts, starting from the simple interpolating methods [15–17]. The proposal by Liu and McMillan [18] employed 'local' (neighboring markers) low-dimensional least squares models combined with PCA for missing marker reconstruction. A significant group of gap reconstruction proposals is based on the low-rank matrix completion methods. They employ various mathematical tools (e.g., matrix factorization with SVD) for the missing data completion, relying on inter marker correlations. Among the others, these methods are described in the following works [19,20]. Another approach is somewhat related: it is a fusion of several regressions and interpolation methods, which was proposed in [21].

Predicting markers (or joint) position is another concept that is the basis of gap-filling techniques. One such concept is a predictive model by Piazza et al. [22], which decomposes the motion into linear and circular and finds momentary predictors by curve fitting. More sophisticated dynamical models based on the Kalman filter (KF) are commonly applied. Wy and Boulanger [23] proposed a KF with velocity constraints; however, this achieved

moderate success due to drift. A KF with an expectation-maximization algorithm was also used in two related approaches by Li et al.—DynaMMo [24], and BoLeRO [25] (the latter is actually Dynammo with bone length constraints). Another approach was proposed by Burke and Lanesby [26], who applied dimensionality reduction by PCA and then Kalman smoothing for the reconstruction of missing markers.

Another group of methods is dictionary-based. These algorithms recover the trajectories using a dictionary created from previously recorded sequences. They result in satisfactory outcomes as long the specific motion is in the database. They are represented by the works of Wang et al. [27], Aristidou et al. [28], and Zhang and van de Panne [29].

Finally, neural networks are another group of methods used in marker trajectory reconstruction. The task can be described as a sequence-to-sequence regression problem, whereas NN applied for regression has been recognized since the early 1990s in the work of Hornik [30]; hence, NN seems to be a natural choice for the task. Surprisingly, however, they become popular quite late. In the work of Fragkiadaki et al. [31], an encoder–recurrentdecoder (ERD) was proposed, employing long-short term memory (LSTM) as a recurrent layer. A similar approach (ERD) was proposed by Harvey et al. [32] for in-between motion generation on the basis of asmall amount of keyframes. Mall et al. [33] modified the ERD and proposed an encoder–bidirectional-filter (EBF) based on the bidirectional LSTM (BIL-STM). In the work of Kucharenko et al. [34], a classical two-layer LSTM and window-based feed-forward NN (FFNN) were employed. A variant of ResNet is applied by Holden [35] to reconstruct marker positions from noisy data as a ttrajectory reconstruction task. A set of extensions to the plain LSTM were proposed by Ji et al. [36]; they introduced attention (a weighting mechanism) and LS-derived spatial constraints, which result in an improvement in performance. Convolution auto-encoders was proposed by Kaufmann et al. [37].

#### **3. Materials and Methods**

#### *3.1. Proposed Regression Approach*

The proposed approach involves employing various neural networks architectures for the regression task. These are FFNN and three variants of contemporary recursive neural networks—gated recurrent unit (GRU), long-short-term memory (LSTM), and bidirectional LSTM (BILSTM). In our proposal, these methods predict trajectories of lost markers on the basis of a local dataset—the trajectories of neighboring markers.

The proposed utilization procedure of NN differs from the scenario that is typically employed in machine learning. We do not feed the NNs with a massive amount of training sequences in advance to form a predictive model. Instead, we consider each sequence separately and try to reconstruct the gaps in individual motion trajectory on the basis of its own data only. This makes sense as long as the marker motion is correlated and most of the sequence is correct and representative enough. This is the same as for the other common regression methods, starting with the least squares. Therefore, the testing data are the whole 'lost' segment (gap), whereas the training is the remaining part of the trajectory. Depending on the gap sizes, and sequence length used in the experiment, the testing can be between 0.6% (for short gaps and long sequences) and up to 57.1% (for long gaps in short sequences).

The selection of such a non-typical approach requires a justification. It is likely that training the NN models for prediction of marker position in a conventional way, using a massive dataset of mocap sequences, would be able to generalize enough to adjust to different body sizes and motions. However, it will be tightly coupled with the marker configuration, not to mention the other actors, such as animals. The other issue is obtaining such a large amount of data. Despite our direct access to the lab resources, this is still quite a cumbersome task, since we believe these might be not enough, especially as the resources available online from various other labs are hardly usable, since they employ different marker setups.

The forecasting of timeseries is a typical problem addressed by RNNs [38]. Usually, numerous training and testing sequences allow for a prediction of the future states of the

modelled system (e.g., power consumption or remaining useful life of devices). A more similar situation, where RNNs are also applied, is forecasting the time series for problems lacking massive training data (e.g., COVID-19 [39]). An analysis of LSTM architectures for similar cases is presented in [40]. However, in these works, the forecast of future values is based on the past values. What makes our case a bit different is the fact that we usually have to predict the value in-the-middle, so the past and future values are available.

#### 3.1.1. Feed Forward Neural Network

FFNN is the simplest neural network architecture. In this architecture, the information flows in one direction, as its structure forms an acyclic directed graph. The neurons are modeled in the nodes with activation functions (usually sigmoid) using the weighted sum of inputs. These networks are typically organized into layers, where the output from the previous layer becomes an input to a successive one. This architecture of networks is employed for regression and classification tasks, either alone or as final stages in a larger structures (such as modern deep NN). The architecture of the NN that we employed is shown in Figure 3. The basic equation (output) of a single—*k*-th artificial neuron is given as:

$$y\_k(\mathbf{x}) = f\left(\sum\_j w\_{jk}\mathbf{x}\_j + b\right),\tag{1}$$

where *xj* is *j*-th input, *wkj* is *j*-th input weight, *b*—a bias value, *f*—is transfer (activation) function. Transfer function depends on the layer purpose; these are typically a sigmoid for hidden layers, threshold, linear, or softmax for final layers (for regression and classification problems, respectively), or others.

**Figure 3.** Schematic of FFNN.

#### 3.1.2. Recurrent Neural Networks

Recurrent neural networks (RNN) are the types of architecture that employ cycles in NN structure; this allows for the consideration of current input value as well as preserving the previous inputs and internal states of NN in memory (and future ones for bidirectional architecture). Such an approach allows for NN to deal with timed processes and to recognize process dynamics, not just static values—it applies to such tasks as a signal prediction or recognition of sequences. Regarding the applicability, aside from classic problem dichotomy (classification and regression), RNN results might need another task differentiation. One must decide whether the task is a sequence-to-one or sequenceto-sequence problem, so the network has to return either a single result for the whole sequence or a single result for each data tuple in sequence. The prediction/regression task is a sequence-to-sequence problem, as demonstrated with RNNs in Figure 4 in different variants—both folded and unfolded, uni- and bi-directional.

**Figure 4.** Usage of recurrent NNs in sequence to sequence task: (**a**) folded, (**b**) unfolded unidirectional variant, (**c**) unfolded bidirectional variant.

At present two types of neuron are predominantly applied in RNN–long short term memory (LSTM) and gated recurrent unit (GRU), of which the former is also applied in bidirectional variant (BILSTM). They evolved from a plain RNN called 'vanilla', and they prevent vanishing gradient problems when back-propagating errors in the learning process. Their detailed designs are unfolded in Figure 5. These cell types rely on the input information and information from previous time steps, and those previous states are represented in various ways. GRU passes an output (hidden signal *h*) between the steps, whereas LSTM also passes a *h* and internal cell state *C*. These values are interpreted as memory—*h* as short term, and *C* as long term. Their activation function is typical sigmoid, which is modeled with a hyperbolic tangent (tanh), but there are additional elements present in the cell. The contributing components, such as input or previous values, are subject to *'gating'*—their share is controlled by Hadamard product (element-wise product denoted as or ⊗ in diagram) with 0–1 sigmoid function *σ*(*x*) = 1 1+*e*−*x* . The individual *σ* values are obtained by weighted input and state values.

**Figure 5.** LSTM (**left**) and GRU (**right**) neurons in detail.

In more detail, in LSTM, we pass two variables *h*, *C* and have three gates—forget, input and output. They govern how much of the respective contribution passes to further processing. The forget gate (*ft*) decides how much of the past cell internal state (*Ct*−1) is to be kept; the input gate (*it*) controls how much new contribution *C*˜*t* caused by input (*xt*) annd taken into the current cell state (*Ct*). Finally, the output gate (*ot*) controls what part of

activation is based on the cell internal state; (*Ct*) is taken as cell output (*ht*). The equations are as follows:

$$f\_t = \sigma(\mathcal{W}\_f \cdot [\mathbf{x}\_{t\prime} h\_{t-1}] + b\_f),\tag{2}$$

$$\mathbf{i}\_{t} = \sigma(\mathcal{W}\_{i} \cdot [\mathbf{x}\_{t}, \mathbf{h}\_{t-1}] + \mathbf{b}\_{f}),\tag{3}$$

$$
\tilde{\mathbf{C}}\_t = \tanh(\mathcal{W}\_\mathbf{c} \cdot [\mathbf{x}\_{t\prime} h\_{t-1}] + b\_\mathbf{c}),
\tag{4}
$$

$$\mathsf{C}\_{t} = f\_{t} \odot \mathsf{C}\_{t-1} + \mathsf{i}\_{t} \odot \tilde{\mathsf{C}}\_{t} \tag{5}$$

$$\circ o\_l = \sigma(\mathcal{W}\_0 \cdot [\mathbf{x}\_{l'} h\_{t-1}] + b\_f),\tag{6}$$

$$h\_l = o\_l \odot \tanh(C\_l). \tag{7}$$

The detailed schematic of GRU is a bit simpler. Only one signal, hidden (layer output) value (*h* for *hi*), is passed between steps. There are two gates present—the reset gate (*rt*), which controls how much past output (*ht*−1) contributes to the overall cell activation, and the update gate (*ut*), which controls how much current activation (˜ *ht*) contributes to the final cell output.The above are described by the following equations:

$$\mathbf{u}\_{t} = \sigma(\mathbf{W}\_{\mathbf{u}} \cdot \left[\mathbf{x}\_{t}, \mathbf{h}\_{t-1}\right] + b\_{\mathbf{u}}),\tag{8}$$

$$r\_t = \sigma(\mathcal{W}\_u \cdot [\mathbf{x}\_t, h\_{t-1}] + b\_u),\tag{9}$$

$$\tilde{h}\_t = \tanh\left(\mathcal{W}\_h \cdot \left[\mathbf{x}\_{t\prime} r\_t \odot h\_{t-1}\right] + b\_h\right),\tag{10}$$

$$h\_t = (1 - u\_t) \odot h\_{t-1} + u\_t \odot h\_t. \tag{11}$$

#### 3.1.3. Employed Reconstruction Methods

We compared the performance of five architectures of NN—two variants of FFNN and three RNN-FCs based on GRU, LSTM, and BILSTM; the outline of the latter is depicted in Figure 6. The detailed structures and hyperparameters of NNs were established empirically, since there are no strict rules or guidelines. Usually, this requires simulating, with parameters sweeping the domain of feasible numbers of layers and neurons [41]. We shared this approach and reviewed the performance of NN using the test data.

The output is three valued *x*, *y*, *z* vectors, containing reconstructed marker coordinates.

**Figure 6.** Proposed RNN-FC architecture for the regression task.

#### 3.1.4. Implementation Details

The training process was performed using 600 epochs, with the SGDM solver running on the GPU. It involved the whole input sequence with gaps excluded. There was a single instance of sequence in the batch. The sequence parts containing gaps were used as the test data; the remainder was used for training—therefore, the relative size of test part varies between 0.6% and 57.1%. The other parameters are:

We also applied z-score normalization for the input and target data.

Additionally, for comparison, we used a pool of other methods, which should provide nice results for short-term gaps. These are interpolations: linear, spline, modified Akima (makima), piecewise cubic hermite interpolating polynomial (pchip), and the low-rank matrix completion method (mSVD0). All but linear interpolation methods are actually variants of piecewise Hermite cubic polynomial interpolations, which differ in the details of how they compute interpolant slopes. Spline is a generic method, whereas pchip tries to preserve shape, and makima avoids overshooting. However, mSVD [42] is an iterative method decomposing motion capture data with SVD and neglecting the least significant part of the basis transformed signal, reconstructing the original data with replacing missing values using reconstructed ones. The procedure finishes when convergence is reached. We implemented the algorithm, as outlined in [24].

The implementation of methods and experiments was carried out in Matlab 2021a using its implementations of numerical methods and deep learning toolbox.

#### *3.2. Input Data Preparation*

Constructing the predictor for certain markers, we obtained the locations from all the sibling markers and a single parent one, as they are organized within an FBM structure. For *j*-th marker (*Xj* = [*xj*, *yj*, *zj*]), we consider parent (*Xp*) and sibling markers (*Xs*1, ... , *XsL*). To form an input vector, we take two of their values—one for the current moment and with one sample lag. The other variants with more lags or values raised to the higher powers were considered, but after preliminary tests, we neglected them since they did not improve performance.

Each input vector *T*, for the moment *n*, is quite long and is assembled of certain parts, as given below:

$$T(n, \star) = \begin{bmatrix} \overbrace{\begin{bmatrix} \underbrace{\text{current and former values of parent mark } (p)}\_{\text{current and four mar value of}} \end{bmatrix}}^{\text{current and form of parent mark } (p)}\_{\text{current and four mar value of first sliding } s\_1} \\ \overbrace{\begin{bmatrix} \underbrace{\text{x\_{s1}}(n), y\_{s1}(n), z\_{s1}(n), x\_{s1}(n-1), y\_{s1}(n-1), z\_{s1}(n-1))} \\ \overbrace{\begin{bmatrix} \underbrace{\text{x\_{s1}}(n), y\_{s1}(n), z\_{s1}(n), x\_{s1}(n-1), y\_{s1}(n-1), z\_{s1}(n-1)} \\ \overbrace{\begin{bmatrix} \underbrace{\text{x\_{s1}}(n), y\_{s1}(n), z\_{s1}(n), x\_{s1}(n-1), y\_{s1}(n-1), z\_{s1}(n-1) \end{bmatrix}} \end{bmatrix}}^{\text{current and four mar value of last sliding } s\_1} \end{bmatrix}}\_{\text{current and four mar value of last sliding } s\_1}. \end{bmatrix} \tag{12}$$

Finally, the input and output data are z-score standardized—zero centered and standard deviation scaled to 1, since such a step notably improves the final results.

#### *3.3. Test Dataset*

For testing purposes, we used a dataset (Table 1) acquired for professional purposes in the motion-capture laboratory. The ground truth sequences were obtained at the PJAIT human motion laboratory using the industrial-grade Vicon MX system. The system capture volume was 9 m × 5 m × 3 m. To minimize the impact of external interference such as

infrared interference from sunlight or vibrations, all windows were permanently darkened and cameras were mounted on scaffolding instead of tripods. The system was equipped with 30 NIR cameras manufactured by Vicon: MX-T40, Bonita10, Vantage V5 wth 10 pieces of each kind.

During the recording, we employed a standard animation pipeline, where data were obtained with Vicon Blade software using a 53-marker setup. The trajectories were acquired at 100 Hz and, by default, they were processed in a standard, industrial-quality way, which includes manual data reviewing, cleaning and denoising, so they can be considered distortion-free.

Several parameters for the test sequences are also presented in Table 2. We selected these parameters as one could consider them to potentially describe prediction difficulty. They are various, and based on different concepts such as information theory, statistics, kinematics, and dynamics, but all characterize the variability in the Mocap signal. They are usually the average value per marker, except for standard deviation (std dev), which reports value per coordinate.

**Table 1.** List of mocap sequence scenarios used for the testing.

Two non-obvious measures are enumerated: monotonicity and complexity. The monotonicity indicates, on average, the extent to which the coordinate is monotonic. For this purpose, we employed an average Spearman rank correlation, which can be described as follows:

$$monotonicity = \frac{1}{M} \sum\_{m=1}^{M} \text{corr}(\text{rank}(X\_i), 1 \dots N), \tag{13}$$

where *Xm* is *m*th coordinate, *M* is number of coordinates, *N* is sequence length.

Complexity, on the other hand, is how we estimate the variability of poses in the sequence. For that purpose, we employed PCA, which identifies eigenposes as a new basis for the sequence. The corresponding eigenvalues describe how much of the overall variance is described by each of the eigenposes. Therefore, we decided to take the remainder of the fraction of variance described by the sum of the five largest eigenvalues (*λi*) as a term describing how complex (or rather simple) the sequence is—the simpler the sequence, the more variance is described, with a few eigenposes. Therefore, our complexity measure is simply given as:

$$\text{Complexity} = 1 - \sum\_{i=1}^{5} \lambda\_i / \sum\_{i=1}^{M} \lambda\_{i\prime} \tag{14}$$

where *M* is a number of coordinates.

**Table 2.** Input sequence characteristics.

#### *3.4. Quality Evaluation*

The natural criterion for the reconstruction task is root mean square error (RMSE), which, in our case, is calculated only for the time and marker, where the gaps occur:

$$\text{RMSE} = \sqrt{\frac{1}{|\mathcal{W}|} \sum\_{i \in \mathcal{W}} (\hat{X}\_i - X\_i)^2},\tag{15}$$

where *W* is a gap map, logically indexing locations of gaps, *X*ˆ is a reconstructed coordinate, *X* is the original coordinate.

Additionally, we calculated RMSEs for individual gaps. Local RMSE is a variant of the above formula, and simply given as:

$$\text{RMSE}\_{\mathbf{k}} = \sqrt{\frac{1}{|w\_k|} \sum\_{i \in \mathcal{w}\_k} (\hat{X}\_i - X\_i)^2} \tag{16}$$

where *wk* ⊂ *W* is a single gap map logically indexing the location of *k*-th gap, *X*ˆ is reconstructed coordinate, *X* is original coordinate. *RMSEk* is intended to reveal variability in reconstruction capabilities; hence, we used it to obtain statistical descriptors—mean, median, mode, and quartiles and interquartile range.

A more complex evaluation of regression models can be based on infromation criteria. These quality measures incorporate squared error and a number of tunable parameters, as they were designed by searching for a tradeoff between the number of tunable parameters and the obtained error. The two most popular ones are Bayesian Information Criterion (BIC) and Akaike Information Criterion (AIC). BIC is calculated as:

$$\text{BIC} = n \log(MSE) + p \log(n), \tag{17}$$

whereas AIC formula is as follows:

$$\text{AIC} = n \log(MSE) + 2p,\tag{18}$$

where: mean squared error MSE = RMSE2, *n* is a number of testing data, *p* is a number of tunable parameters.

#### *3.5. Experimental Protocol*

During the experiments, we simulated gap occurrence in perfectly reconstructed source sequences. We simulated gaps of different average lengths—10, 20, 50, 100 and 200 samples (0.1, 0.2, 0.5, 1, and 2 s, respectively). The assumed gap sizes were chosen to represent situations of various levels of difficulty, from short-and-simple to difficult ones, when gaps are long. For every gap length, we performed 100 simulation iterations, where the training and testing data do not intermix between simulation runs. The steps performed in every iteration are as follows:

We report the results as RMSE and descriptive statistical descriptors for *RMSEk* for every considered reconstruction technique. Additionally, we verified the correlation between RMSE and the variability descriptors for sequences. It is intended to reveal what are the sources of difficulties in predicting the marker trajectories.

#### Gap Generation Procedure

The procedure of gap contamination, which was employed, introduces distortions into the sequences in a controlled way. The parameter characterizing the experiment is an average-length number of occurrences of gaps. the sequence of operations distorting the signal is as follows: at first, we draw moments to contaminate, then select a random marker. The duration of distortions and intervals is a Poisson process, an average length of distortion set-up according to the considered gap length in the experiment, whereas the interval length results from the sequence length and number of intervals, which, for two gaps per sequence, are three—ahead of the first gap, in-between, and after the second gap.

#### **4. Results and Discussion**

The section comprises two parts. First, we present RMSE results; they illustrate the performance of each of the considered gap reconstruction methods. The second part is the interpretation of results, searching for the aspects of Mocap sequence that might affect the resulting performance.

#### *4.1. Gap Reconstruction Efficiency*

The detailed numerical values are presented in Table 3 for the first sequence as an example. In the table, we also emphasize the best result for each measurement of gap size. Forclarity, the numerical outcomes of the experiment are only presented in this chapter with representative examples. To see the complete set of results in the tabular form, please refer to Appendix A. The complete results for the gap reconstruction are also demonstrated in a visual form in Figure 7. Additionally, the zoomed variant of the fragments of the plot (dash square annotated) for gaps 10–50 are presented in Figure 8.

The first observation, regarding the performance measures, is the fact that the results are very coherent, regardless of which measure was used. This is shown in Figure 7, where all the symbols coherently denote statistical descriptors scale. It is also clearly visible in the values emphasized in Table 3, where all measures but one (mode) indicate the same best (smallest) results. Hence, we can use a single quality measure; in our case, we assumed RMSE for further analysis.

Analyzing the results for several sequences, various observations regarding the performance of the considered methods can be noted. These are listed below:

values; the 'running' sequence has similar values for these, but FFNNlin perform the best in this case, so the kinematic/dynamic parameters should not be considered.

**Figure 7.** Results for most of the quality measures for all the test sequences. Bars denote *RMSE*; for *RMSEk*: denotes mean value, × denotes median, ◦ denotes mode, whiskers indicate IQR; standard deviation is not depicted here; dash-outlined areas are zoomed in Figure 8.

**Figure 8.** Results of the most of the quality measures for all the test sequences—zoomed variant for gaps 10, 20, and 50. Bars denote *RMSE*; for *RMSEk*: denotes mean value, × denotes median, ◦ denotes mode, whiskers indicate IQR; standard deviation is not depicted here.

**Table 3.** Quality measures for the static (No. 1) sequence.

Looking at the results of various NN architectures, it might be surprising that the sophisticated RNNs often returned worse results than relatively simple FFNN, especially for relatively long gaps. Conversely, one might expect that RNNs would outperform other methods, since they would be able to model longer-term dependencies in the motion. Presumably, the source of such a result is in the limited amount of training data, which, depending on the length of the source file, varies between hundreds and thousands of registered coordinates. Therefore, solvers are unable to find actually good values for a massive amount of parameters—see Table 4 for the formulas and numbers of learnable parameters for an exemplary case when input comprises 30 values—coordinates of four siblings and a parent at current and previous frames.

An obvious solution to such an issue would be increasing the training data. We could achieve this by employing very long recordings or by using numerous recordings. In the former, it would be difficult to achieve long enough recordings; the latter is different from the case which we try to address, where we only obtain a fresh mocap recording and reconstruct it with the minimal model given by FBM. Training the predictive model in advance with a massive amount of data is, of course, an interesting solution, but would cost the generality. For every marker configuration, a separate set of predicting NNs would need to be trained, so the result would only be practical for standardized body models.

Considering the length of the training sequences, its contribution to the final results seems far less important than other factors, at least within the range of considered cases. The analysis of its influence is illustrated in Figure 9. Since the MSE results are entangled, we employed two additional information criterions, Akaike Information Criterion (AIC), Bayesian Information Criterion (BIC), which disentangle the results by accounting for the

number of trainable parameters. For every sequence and every NN model, we obtain a series of five results, which decrease, as the training sequence grows longer when we have shorter gaps (i.e., the annotated quintuple in the Figure). Analyzing the results in Figure 9, it is most convenient to observe this in the AIC/BIC plots since, for each model, the number of parameters remains the same (Table 4), so we can easily compare the results of the testing sequences. The zoomed versions (to the right) reveal differences at appropriate scales for the RNN results.

Lookng at the reults, we observe that, regardless the length of the training sequence, the MSE (AIC/BIC) of the NN model remains at the same order of magnitude—this is clearly visible in the Figure, where we have very similar values for each gap size for variable sequences (represented as different marker shapes) for each of the NN types (represented by a color). The most notable reduction in the error is probably observed with the increased sequence length, when the sequence (Seq. 1—static) is several folds longer than the others. However, we cannot observe this difference for shorter sequences in our data, with notably different lengths (e.g., walking—running). The quality of prediction could be likely improved if the recordings were longer, but, in everyday praxis, the length of the motion caputre sequences is only minutes, so one should not expect the results for RNN data to be notably improved compared to those for FFNN.

The observations hold for both FFNN models and all RNNs. These ambiguous outcomes confirm the results shown in [40], where the quality of results does not depend on the length of the training data in a straightforward way.

**Table 4.** List of mocap sequence scenarios used for the testing.

#### *4.2. Motion Factors Affecting Performance*

In this section, we try to identify the correlation in which features (parameters) of the input sequences relate to the performance of gap-filling methods. The results presented here are concise; we only present and discuss the most conclusive results. The complete tables containing correlation values for all gap sizes are presented in Appendix B.

Foremost, a generalized view into the correlation between gap-filling outcomes and input sequence characteristics is given in Table 5. It contains Pearson correlation coefficients (CC) between RMSE and input sequence characteristic parameters; the values are Pearson CCs, averaged across all the considered gap sizes. Additionally, for the interpretation of the results, in Table 6, we provide CCs between RMSE and the descriptive parameters for the whole sequences for all the test recordings.

**Figure 9.** Influence of training sequence length on the quality of obtained results for NN methods: Akaike Information Criterion (AIC), Bayesian Information Criterion (BIC) and MSE.

Knowing that correlation, as a statistical measure, makes little sense for a sparse dataset, we treat it as a kind of measurement of co-linearity between the measures. However, for part of the parameters, the (high) correlation values are connected, with quite satisfactory low *p*-values; these are given in Appendix B.

**Table 5.** Correlation between RMSE and sequence parameters (averaged for all gap sizes).

**Table 6.** Correlation between sequence parameters.

Looking into the results in Table 5, we observe that all the considered sequence parameters are related, to some extent, to RMSE. However, for all the gap-filling methods, we identified two key parameters that have higher CCs than the others. These are acceleration and monotonicity, which seem to be promising candidate measures for describing the susceptibility of sequences to the employed reconstruction methods.

Regarding inter-parameter correlations in Table 6, we can observe that most of the measures are correlated with each other. This is expected, since kinematic/dynamic parameters are connected with the location of the markers over time, so values such as entropy, position standard deviation, velocity, acceleration, and jerk are correlated (for the derivatives, the smaller the difference in the derivative order, the higher the CCs).

On the other hand, the two less typical measures, monotonicity and complexity, are different; therefore, their correlation with the other measures is less predictable. Complexity appeared to have a notable negative correlation with most of the typical measures. Monotonicity, on the other hand, is more interesting. Since it is only moderately correlated with remaining measures, it still has quite a high CC, with RMSEs for all the gap reconstruction methods. Therefore, we can suppose this describes an aspect of the sequence that is independent of the other measures, which is related to susceptibility to the gap reconstruction procedures.

#### **5. Summary**

In this article, we addressed the issue of filling the gaps that occurred in the mocap signal. We considered this to be a regressive problem and reviewed the results of several NN-based regressors, which were compared with several interpolation and low-rank matrix completion (mSVD) methods.

Generally, in the case of short gaps, the interpolation methods returned the best results, but since the gaps became longer, part of the NNs gained an advantage. We reviewed

five variants of neural networks. Surprisingly, the tests revealed that simple linear FFNNs, using momentary (current and previous sample) and local (from neighboring markers) coordinates as input data, outperformed quite advanced recurrent NNs for the longer gaps. For the shorter gaps, RNNs offered better results, but all the NNs were outperformed by interpolations. The boundary between 'long' and 'short' terms are gaps of 50 samples long. Finally, we were able to identify which factors of the input mocap sequence influence the reconstruction errors.

The approach to the NNs given here does not incorporate skeletal information. Instead, the kinematic structure is based on the FBM framework and all the predictions are performed with the local data, as obtained from FBM. Currently, none of the analyzed approaches considered body constraints such as limb length or size, but we can easily obtain such information from the FBM model. We plan to apply this as an additional processing stage in the future. In the future, we plan to test more sophisticated NN architectures, such as combined LSTM convolution, or averaged multiregressions.

**Supplementary Materials:** The following are available at https://www.mdpi.com/1424-8220/21/1 8/6115/s1, The motion capture sequences.

**Author Contributions:** conceptualization, P.S.; methodology, P.S., M.P.; software, P.S., M.P.; investigation, P.S.; resources, M.P.; data curation, M.P.; writing—original draft preparation, P.S.; writing—review and editing, P.S., M.P.; visualization, P.S. All authors have read and agreed to the published version of the manuscript.

**Funding:** The research described in the paper was performed within the statutory project of the Department of Graphics, Computer Vision and Digital Systems at the Silesian University of Technology, Gliwice (RAU-6, 2021). APC were covered from statutory research funds. M.P. was supported by grant no WND-RPSL.01.02.00-24-00AC/19-011 funded by under the Regional Operational Programme of the Silesia Voivodeship in the years 2014–2020.

**Institutional Review Board Statement:** Not applicable.

**Informed Consent Statement:** Not applicable.

**Data Availability Statement:** The motion capture sequences are provided as Supplementary Files accompanying the article.

**Acknowledgments:** The research was supported with motion data by Human Motion Laboratory of Polish-Japanese Academy of Information Technology.

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

#### **Abbreviations**

The following abbreviations are used in this manuscript:

#### **Appendix A. Performance Results for All Sequences**

**Table A1.** Quality measures for the walking (No. 2) sequence.

**Table A2.** *Cont.*

**Table A3.** Quality measures for the sitting (No. 4) sequence.

**Len FFNNlin FFNNtanh LSTM GRU BILSTM LIN SPLINE MAKIMA PCHIP mSVD** 20 RMSE 37.101 48.909 51.388 50.842 51.274 72.745 51.478 59.839 59.857 72.745 mean(RMSE*k*) 31.439 41.811 44.331 43.711 44.219 66.280 44.321 54.030 54.156 66.280 median(RMSE*k*) 26.422 36.792 40.178 39.257 40.099 71.201 39.395 55.235 54.311 71.201 mode(RMSE*k*) 1.783 2.342 2.592 2.372 2.558 0.972 2.819 0.875 0.912 0.972 stddev(RMSE*k*) 20.198 25.924 26.514 26.496 26.480 30.443 26.659 26.183 26.001 30.443 iqr(RMSE*k*) 22.947 30.188 29.510 29.617 29.241 37.209 29.316 29.572 28.094 37.209

**Table A3.** *Cont*.

**Table A4.** Quality measures for the boxing (No. 5) sequence.

**Table A5.** Quality measures for the falling (No. 6) sequence.

**Table A5.** *Cont.*

#### **Appendix B. Correlations between RMSE an Sequence Parameters**

**Table A6.** Correlation between RMSE and entropy of input sequence.

**Table A7.** Correlation between RMSE and standard deviation of input sequence.

**Table A7.** *Cont*.

**Table A8.** Correlation between RMSE and velocity of input sequence.

**Table A9.** Correlation between RMSE and acceleration of input sequence.

**Table A10.** Correlation between RMSE and jerk of input sequence.

**Table A11.** Correlation between RMSE and monotonicity of input sequence.

**Table A12.** Correlation between RMSE and complexity of input sequence.

#### **References**

## *Article* **Human Action Recognition: A Paradigm of Best Deep Learning Features Selection and Serial Based Extended Fusion**

**Seemab Khan 1, Muhammad Attique Khan 1,\*, Majed Alhaisoni 2, Usman Tariq 3, Hwan-Seung Yong 4, Ammar Armghan 5 and Fayadh Alenezi 5**

**Abstract:** Human action recognition (HAR) has gained significant attention recently as it can be adopted for a smart surveillance system in Multimedia. However, HAR is a challenging task because of the variety of human actions in daily life. Various solutions based on computer vision (CV) have been proposed in the literature which did not prove to be successful due to large video sequences which need to be processed in surveillance systems. The problem exacerbates in the presence of multiview cameras. Recently, the development of deep learning (DL)-based systems has shown significant success for HAR even for multi-view camera systems. In this research work, a DL-based design is proposed for HAR. The proposed design consists of multiple steps including feature mapping, feature fusion and feature selection. For the initial feature mapping step, two pre-trained models are considered, such as DenseNet201 and InceptionV3. Later, the extracted deep features are fused using the Serial based Extended (SbE) approach. Later on, the best features are selected using Kurtosiscontrolled Weighted KNN. The selected features are classified using several supervised learning algorithms. To show the efficacy of the proposed design, we used several datasets, such as KTH, IXMAS, WVU, and Hollywood. Experimental results showed that the proposed design achieved accuracies of 99.3%, 97.4%, 99.8%, and 99.9%, respectively, on these datasets. Furthermore, the feature selection step performed better in terms of computational time compared with the state-of-the-art.

**Keywords:** human action recognition; deep learning; features fusion; features selection; recognition

#### **1. Introduction**

Human action recognition (HAR) emerged as an active research area in the field of computer vision (CV) in the last decade [1]. HAR has applications in various domains including; surveillance [2], human-computer interaction (HCI) [3], video reclamation, and understanding of visual information [4], etc. The most important application of action recognition is video surveillance [5]. Governments use this application for intelligence gathering, reducing crime rate, for security purposes [6], or even crime investigation [7]. The main motivation of growing research in HAR is due to its use in video surveillance applications [8]. In visual surveillance, HAR plays a key role in recognizing the activities of subjects in public places. Furthermore, these types of systems are also useful in smart cities surveillance [9].

Human actions are of various types. These actions can be categorized into two broad classes, namely voluntary actions and involuntary actions [10]. Manual recognition of

**Citation:** Khan, S.; Khan, M.A.; Alhaisoni, M.; Tariq, U.; Yong, H.-S.; Armghan, A.; Alenezi, F. Human Action Recognition: A Paradigm of Best Deep Learning Features Selection and Serial Based Extended Fusion. *Sensors* **2021**, *21*, 7941. https://doi.org/10.3390/s21237941

Academic Editors: Tomasz Krzeszowski, Adam Swito ´ ´ nski, Michal Kepski and Carlos Tavares Calafate

Received: 4 November 2021 Accepted: 25 November 2021 Published: 28 November 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

these actions in real-time is a tedious and error-prone task; therefore, many CV techniques are introduced in the literature [11,12] to serve this task. Most of the proposed solutions are based on classical techniques such as shape features, texture features, point features, and geometric features [13]. A few techniques are based on the temporal information of the human [14], and a few of them extract human silhouettes before feature extraction [15].

Recently, deep learning has shown promising results in the field of computer vision (CV) [16]. Deep learning makes learning and data representation at multiple levels by mimicking the human brain processing [17] to create models. These models consist of multiple processing layers such as convolutional, ReLu, pooling, fully connected, and Softmax [18]. The functionality of a CNN model is to replicate the working of the human brain as it preserves and makes sense of multidimensional information. There exist multiple methods in deep learning, which include encompassing neural networks, hierarchical probabilistic models, supervised learning, and unsupervised learning models [19].

The HAR process is a challenging task as there are a variety of human actions in daily life. In order to tackle this challenge, deep learning models are utilized. The performance of a deep learning model is always based on the number of training samples [20]. In the action recognition tasks, several datasets are publicly available. These datasets include several actions such as walking, running, leaving a car, waving, kicking, boxing, throwing, falling, bending down, and many more.

Recently proposed systems mainly focus on the hybrid techniques; however, they do not focus on minimizing the computational time [21]. This is an important factor as most time surveillance is performed in real-time. Some of the other key challenges of HAR are as follows: (i) Query video sequences resolution is imperative for the recognition of the focal point in the most recent frame. The background complexity, shadows, lighting conditions, and outfit conditions extract irrelevant information using classical techniques of human action, which later results in inefficient action classification; (ii) with automatic activities recognition under multi-view cameras it is difficult to classify the correct human activities. Change in the motion variation captures the wrong activities under the multi-view cameras; (iii) imbalanced datasets impact the learning of a CNN. A CNN model always needs a massive number of training images for learning; and (iv) features extraction from the entire video sequences includes several irrelevant features, affecting the classification accuracy.