The CNN parametric classifier performed with a very high accuracy value, quite close to 100%. We also tried to run inference of this model using a CPU, and we got up to 500 gestures per second on computer with an Intel® Core™ i7-7700 processor ((Intel Corporation, Santa Clara, CA, USA) and 16 GB of RAM. This speed of processing means that inference in real time is possible using the CNN classifier.

The difference between the accuracies of both models tested in this work might occur because the CNN model captures better the distribution underlying the data of the ten gestures that we used here. However, with the increase of the number of training examples, the *k*-NN and DTW model might lead to similar or even better results than the CNN model at the cost of a higher time of processing. However, increasing the number of training examples will increase the time of training of the CNN, but its time of prediction will be fixed since the complexity of the CNN does not change with the increase of the size of the training set. This last feature of the CNN and the results obtained in this work evidence that this model is an interesting option for sign language classification in scenarios with many training examples.

We reported comparisons of our results with those from other relevant works published in the scientific literature. Although a perfect fit among all the works is almost impossible, since there are differences in terms of the types of wearables and signs, and number of signers and signs, the higher accuracies obtained in this work suggest how our approach can be valuable for sign recognition purposes. There are some major differences with respect to other studies. In particular, the number of used sensors, that constitute a complete set for full fingers, hands/arms/forearms joints analysis; the 100 gesture repetitions for each LIS sign from each signer, that constitute a comprehensive set of valuable data, permit us a fair and reliable comparison between two different data analysis approach. Moreover, the data from the sensory glove and the IMUs are collected and synchronized, providing real-time data-fusion from both sensory glove and IMUs.

As a meaningful further consideration, our database was constructed with signs performed by trainees rather than expert native signers, which adds a sort of "robustness" to the tested classification algorithms. Indeed, a non-native speaker typically repeats a specific gesture with minor precision with respect to a native speaker. Thus, the analysis applies to more dispersed values, thus enabling an expected better performance in case of native-speaker better-accuracy gestures (to be demonstrated in a future work).

Further research includes improvements on the *k*-NN and DTW model using an efficient reduction of the data dimensionality. In this way, better and faster results might be obtained, which in turn may also allow us to recognize more gestures for real-time applications. Finally, both models need to be tested at classifying more classes for different sign languages and for the recognition of consecutive sentences.

**Author Contributions:** Conceptualization, G.S. and M.E.B.; methodology, G.S., P.C., J.Z., and M.E.B.; software, P.C., M.R., and J.Z.; validation, G.S., P.C., and M.E.B.; formal analysis, P.C. and J.Z.; investigation, G.S., M.R., V.E., J.Z., and M.E.B.; resources, G.S. and M.E.B.; data curation, G.S., P.C., M.R., V.E., J.Z., and M.E.B.; writing—original draft preparation, G.S., P.C., and M.E.B.; writing—review and editing, G.S. and M.E.B.; visualization, G.S. and M.E.B.; supervision, G.S. and M.E.B.; project administration, G.S. and M.E.B.; funding acquisition, G.S. and M.E.B. All authors have read and agreed to the published version of the manuscript.

**Funding:** Authors gratefully acknowledge the financial support provided by Escuela Politécnica Nacional for the development of the research project PIJ-16-13 "Clasificación de señales electromiográficas del brazo humano usando técnicas de reconocimiento de patrones y machine learning".

**Conflicts of Interest:** The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.

### **References**

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

### *Article*

## **Analysis of Control Characteristics between Dominant and Non-Dominant Hands by Transient Responses of Circular Tracking Movements in 3D Virtual Reality Space**

### **Wookhyun Park 1, Woong Choi 2,\*, Hanjin Jo 1, Geonhui Lee 1 and Jaehyo Kim 1,\***

Received: 28 May 2020; Accepted: 18 June 2020; Published: 19 June 2020

**Abstract:** Human movement is a controlled result of the sensory-motor system, and the motor control mechanism has been studied through diverse movements. The present study examined control characteristics of dominant and non-dominant hands by analyzing the transient responses of circular tracking movements in 3D virtual reality space. A visual target rotated in a circular trajectory at four different speeds, and 29 participants tracked the target with their hands. The position of each subject's hand was measured, and the following three parameters were investigated: normalized initial peak velocity (*IPV2*), initial peak time (*IPT2*), and time delay (*TD2*). The *IPV2* of both hands decreased as target speed increased. The results of *IPT2* revealed that the dominant hand reached its peak velocity 0.0423 s earlier than the non-dominant hand, regardless of target speed. The *TD2* of the hands diminished by 0.0218 s on average as target speed increased, but the dominant hand statistically revealed a 0.0417-s shorter *TD2* than the non-dominant hand. Velocity-control performances from the *IPV2* and *IPT2* suggested that an identical internal model controls movement in both hands, whereas the dominant hand is likely more experienced than the non-dominant hand in reacting to neural commands, resulting in better reactivity in the movement task.

**Keywords:** circular tracking movement; motor control; virtual reality; dominant hand; transient response

### **1. Introduction**

Researchers in various academic areas have studied human movement to better understand human motor control mechanisms [1–25]. In particular, the human body produces a single movement through the combined coordination of multiple joints and muscles [18,26], while the human brain controls physiological movements by receiving sensory feedback and transmitting neural commands. As such, many individual movements involve a variety of motor control behaviors [19–48].

Various target-tracking movements have been studied using a multi-joint upper-limb to analyze human visuo-motor systems [26–48]. Several studies have used a visual target moving in a straight line [44–47] or a sinusoidal waveform [26–29] to examine motor characteristics on a 1-DOF arm movement, and many have used target-tracking on a circular trajectory to investigate the velocity-dependent behaviors of upper-limb movement [33–39]. The control characteristics of the visuo-motor system have also been examined to ascertain their contribution to intermittency [28–32] and proactiveness in response to a predictable target [26,39–41].

Handedness and hand usage are another important aspect of upper-limb movement [47–53]. Several efforts have been made in the physiological field to elucidate which factors determine the preferred or dominant hand (*DH*) [52,53]. Functional comparisons between the *DH* and non-*DH* (*NDH*), as well as between right- and left-handed individuals, have investigated the control characteristics of upper-extremity movement [47–51]. These studies revealed that control performance differed between the hands when the given movement task required a high level of accuracy or dexterity [47,48], although handedness does not directly cause the performance differences between the hands [49,50].

Recently, we implemented a system that measures the Cartesian movement of an end-effector within a three-dimensional virtual reality environment, enabling quantitative evaluation of human hand movement during various movements, such as reaching and tracking [37,38]. In one study, we examined a visual target rotated along a specific circular trajectory, as well as subjects' tracking performances under monocular and binocular vision [37]. In another, we analyzed how the human visuo-motor system controls temporo-spatial parameters during circular tracking movements [38].

Circular target-tracking tasks have been distinguished from other movement tasks because the continuous motion allows researchers to analyze the velocity properties of upper-limb end-point movement [25,32,33]. Previous studies have compared subjects' control of stable behaviors at different target speeds during circular tracking tasks [37,38].

To comprehend the control mechanisms of the human upper-extremity, researchers must examine the initial behaviors in a tracking movement. In control engineering, the time-domain response of a controlled system is classified into a transient response and a steady-state response [54]. For instance, assuming a step signal is given to a system, the rise time and overshoot are frequently evaluated in the performance analysis of the system's transient response. Steady-state error is often determined to examine the stability of a control system's steady-state response. Considering the human visuo-motor system as a controlled system, the transient responses of the target-tracking movement under various target speeds are thought to indicate the velocity-dependent motor control mechanism.

Therefore, in the present study, velocity-dependent motor control characteristics were examined using the transient responses of the circular tracking movements of the *DH* and *NDH* in a 3D space. Temporal behaviors in response to a moving target were analyzed using three parameters: the normalized magnitude of the hand's initial peak velocity on the 2D plane (*IPV2*), the corresponding initial-peak time (*IPT2*), and the calculated time delay of the hand (*TD2*). Participants used their *DH* and *NDH* to track a visual target rotating in a circular trajectory at four different speeds.

### **2. Materials and Methods**

### *2.1. Subjects and Experimental Setup*

The present study involved 29 subjects: 19 males and 10 females (Table 1). Their mean age was 24.67 ± 3.1 years, and 26 of them were right-handed while the others were left-handed. All subjects had normal or corrected-to-normal vision and decent physical condition without any medical history regarding their upper limbs. The subjects had no experience of similar studies or any relevant measurement devices utilized in this experiment, and all of them provided written consent prior to participation in the research. All experiments were performed in compliance with relevant guidelines and regulations. The protocol was approved by the ethics committee of the National Institute of Technology, Gunma College.

**Table 1.** Characteristics of 29 Subjects.

The experiment aimed to investigate visuo-motor control characteristics at different target speeds at the beginning of visual guided tracking tasks using a virtual-reality (VR) system. Such VR systems must (1) provide an immersive 3D audio-visual environment, (2) visualize preprogramed 3D objects and operate them in real-time, and (3) record and display the movement of the user's hand in the environment [37]. HTC Vive was used to give visually guided tracking tasks to the subjects and to display the position of their hands in real time. The head-mounted display (HMD) offered 2160 × 1200-pixel graphic resolution at a refresh rate of 90 Hz. The compatible controller weighed 200 g and was virtually illustrated as 20 cm long with a white stick in the VR space.

An experimental chair designed for the research was placed in the center of the camera-viewed space. The subjects sat on this chair, and two belts fixed on each side were used to fasten their upper bodies to prevent undesired shoulder movements.

### *2.2. Experimental Procedures*

The subjects were fully informed about the research and safely prepared for the experiment. To start with, they were instructed to read the written notice thoroughly and given verbal instructions by the experimenter. They were then asked to fill in a consent form and assigned to the experimental chair. After they were seated, they were carefully fastened onto the chair and received a VR face-protection cover. They were then equipped with the HMD and grasped a hand-held controller (Figure 1A). To become acquainted with and to practice interaction within the provided VR environment, individual subjects were assigned 20 min of a familiarization session with basic VR programs embedded in the HTC Vive system.

**Figure 1.** Experimental procedure for the circular tracking experiment. (**A**) Subject sitting on an experimental chair and holding a hand-held controller to conduct the target tracking movement with the virtual reality (VR) head-mounted display on. To restrict upper-body interference, belts are crossed on the subject's chest. The target (red ball) is only visible in the 3D VR environment, while the tracer (yellow ball) indicates the current 3D location of the controller. The orbit of the target is invisible to the subject. (**B**) The sizes and trajectory of the balls. The target must move along a virtual trajectory with a diameter of 300 mm. They are instructed to follow the target with the tracer. (**C**) The target moves within the frontal plane and the velocity of the tracer is examined.

Once the subjects sufficiently performed the session, a brief checkup was given to verify if the subjects find any inconvenience prior to the experiment, and the experimental program was initiated. The HMD showed a target with a red, ball-shaped sphere and a tracer with a yellow, ball-shaped sphere that had a white stick attached to it. Both had a radius of 15 mm but differed in size depending on their depths (Figure 1B). The background was black (Figure 1C).

The controller was custom calibrated to the arms of each individual subject. Specifically, the subject initially placed the controller immediately beneath their jaw and pulled the trigger. The experimenter then told the subject to reach their arm straight forward and release the trigger after full reaching. The tracer was continuously displayed on the HMD, and the target appeared after the trigger was released. The target was set at a position of 200 mm in the positive direction on the *Z*-axis and 150 mm in the positive direction on the *Y*-axis from the position of the tracer. This process was executed when the subject changed their hand to grab the controller.

The subjects were asked to place the tracer in the center of the target to verify whether the calibration was appropriate. They were then given verbal corrections to ensure they were aware of the initial position of the target. No physical adjustment was applied to the subjects.

The subjects were then directed to maintain the position of the tracer. They were also informed that four "beep" sounds would be played to indicate a 3-s countdown and that the subjects should start to track the target as closely as possible using the tracer after the 4th beep sound, regardless of the target speed. When they accurately placed the tracer and maintained their posture, the experimenter notified the start of a task and ran the program. The speaker on the HMD produced the beep sounds, and the subject started tracking immediately after hearing the last sound.

### *2.3. Movement Tasks*

The experiment consisted of two main tasks. The first was to track the target with the tracer using the *DH*. The target, displayed at the initially calibrated position (0, 150, 0 mm), moved in a circular trajectory in the clockwise direction on the frontal (*X*-*Y*) plane. The radius of the orbit was 150 mm, and the target revolved six times at a constant speed. The target speeds were set at 0.250 π, 0.500 π, 1.000 π, and 1.500 π rad/s along the trajectory. Only one trial was carried out at each speed, and the subjects were told that the target would move faster in the next trial. A break of 30 s was given after each trial.

The second task was to perform the same type of tracking movement using the *NDH*. At the beginning of the experiment, each subject was randomly assigned which hand would start the tracking movement: *DH* or *NDH*. The controller was re-calibrated for each hand.

### *2.4. Data Analysis*

Real-time positions of the tracer in the Cartesian coordinates (*X*,*Y*,*Z*) were used for data analysis. The sampling frequency was 90 Hz and the transient response of the circular tracking motion was examined using the data measured after the target started moving. As a transient response of a controlled system illustrates behavior to instantaneous changes in its reference input and a process of reaching a steady-state response, so the initial performance of the tracer indicated a transient response of how the subject instantly started the target-tracking movement.

Three parameters were chosen to analyze the performance of the transient response. The first was the *IPV2*, the initial peak velocity of the tracer on the frontal plane. The measured *X* and *Y* values were numerically differentiated and computed with respect to the speed of the tracer. Considering the target speed as a referenced input, the *IPV2* could be interpreted as an overshoot of the transient response in a controlled system. The *IPV2* was normalized to the corresponding target speed to allow comparisons with other *IPV2* values at different target speeds. The *IPV2* was calculated as a percentage.

$$v\_{x,trc}[t] = \begin{cases} \frac{x\_{trc}[t] - x\_{trc}[t-1]}{T\_s} & \text{for } t > 1\\ 0 & \text{for } t = 1 \end{cases} \tag{1}$$

$$w\_{y, trc}[t] = \begin{cases} \frac{y\_{trc}[t] - y\_{trc}[t-1]}{T\_s} & \text{for } t > 1\\ 0 & \text{for } t = 1 \end{cases} \tag{2}$$

$$v\_{xy,trc}[t] = \sqrt{v\_{x,trc}[t]^2 + v\_{y,trc}[t]^2} \tag{3}$$

$$\left.v\_{xy,trc}^{p\text{cnk}} = \max(v\_{xy,trc}[t])\right|\_{t=1}^{n\_{lr}} \tag{4}$$

$$
\omega\_{xy, trg} = r\_{traj} \times a\_{trg} \tag{5}
$$

$$IPV2 = \frac{v\_{xy, trc}^{pank}}{v\_{xy, trg}} \times 100\tag{6}$$

Constant *nhr* denotes the length of data measured at a sample rate of 90 Hz while the target rotated through half of the first orbit. The *nhr* values at each speed were 360 (0.250 π rad/s), 180 (0.500 π rad/s), 90 (1.000 π rad/s), and 60 (1.500 π rad/s). The abbreviations *trg* and *trc* represent the target and tracer, respectively. The velocity of the target on the frontal plane, *vxy,trg*, was decided by the target's angular speed ω*trg* which was one of 0.250 π, 0.500 π, 1.000 π, and 1.500 π rad/s. The radius of the target's trajectory *rtraj* was fixed to 150 mm and the *vxy,trg* was considered as a constant because the target rotated the circular trajectory at a constant velocity during a single trial.

The second parameter was *IPT2*, the initial peak time, which is the time at which *IPV2* occurred.

$$IPT2 = t \left[ v\_{xy, trc}^{pank} \right]\_{t=1}^{n} \tag{7}$$

The last parameter was *TD2*, the time delay or reaction time of the tracer on the frontal plane. In order to examine the transient response, it is essential to know the starting moment of the response. Other studies could easily distinguish the moment of when the subject actually started to move since they employed measurement devices with 1- or 2-DOF [26–48]. Such devices were fixed on a certain spot or plane, and the measured data did not fluctuate for a stationary movement, allowing easy detection of positional changes of the tracer. The VR environment of this experiment, however, did not physically restrict any movement of our subjects. In fact, they placed their hands in the air without any structural support. Though the subjects were told to stay still, physiological factors such as cardiac impulses or a hand tremor may cause positional fluctuation of the hands. These unintentional movements hinder the detection of the starting moment of the response. By identifying features of the involuntary movements, the transient response of the target tracking movement can be discriminated from the unintentional fluctuation.

In statistics, one standard deviation of a normal distribution denotes how much difference each data point has from its mean value, expressing the broadness of the data. In a normal distribution, 68.2% of the data lies within one standard deviation of the mean, and three standard deviations cover 99.6% of the sample.

Applying this theory to our experiment, one standard deviation of the tracer during a 3-s countdown (*STDtrc*) represented a statistic feature of the involuntary movements prior to the tracking movement. That is, *STDtrc* could be referred to as a magnitude of the hand's tremor or how widely the tracer was fluctuated irrespective of the subject's intention. If the displacement of the tracer from its initial position was greater than three standard deviations, this could be interpreted as the subject moves the tracer with clear intention. The *TD2* value expressed the moment at which the tracer's displacement from its initial position became more than three times that of the *STDtrc*.

$$disp\_{trc}[t] = \sqrt{(\mathbf{x}\_{trc}[t] - \mathbf{0})^2 + (y\_{trc}[t] - 15\mathbf{0})^2} \text{ for } -270 \le \mathbf{t} \le -1 \tag{8}$$

$$disp\_{\rm trc}^{\rm man} = \frac{\sum\_{t=-270}^{-1} disp\_{\rm trc}[t]}{n} \tag{9}$$

$$STD\_{\rm trc} = \sqrt{\frac{\sum\_{t=-270}^{-1} \left(disp\_{\rm trc}[t] - disp\_{\rm trc}^{\rm man}\right)^2}{n-1}} \tag{10}$$

$$d\_{xy, trc}[t] \text{ [mm]} = \sqrt{\left(\mathbf{x}\_{trc}(t) - \mathbf{x}\_{trc}^{init}\right)^2 + \left(y\_{trc}(t) - y\_{trc}^{init}\right)^2} \text{ for } t > 0 \tag{11}$$

$$TD2 = \min\{t[d\_{xy, trc}(t) \ge 3 \times STD\_{trc}]\Big|\_{t=1}^{n\_{lr}}\tag{12}$$

*disptrc* represented a displacement of the tracer from the target's position (0, 150, 0 mm) on the frontal plane. Since the sampling frequency was 90 Hz, 270 position data were measured during the 3-s countdown (the time indices ranged from −270 to −1 to distinguish from the data measured after the countdown). Equation (8) indicated that displacements were calculated for the three seconds before the start. *dispmean trc* in Equation (9) is a mean value of the collected *disptrc*, which is used for computing the value of *STDtrc* in Equation (10). After the 4th beep, the subject may initiate to move the tracer, but the initial position of the tracer may differ from that of the target, although the subjects were told to place the tracer in the center of the target prior to the experiment. The tracer is on the frontal plane (*xinit trc* , *yinit trc* ) at the start of each trial. Displacements after the start were calculated as *dxy*,*trc* as shown in Equation (11), and the moment that the displacement becomes greater than triple of *STDtrc* was considered as *TD2*.

All these parameters were calculated from data measured for two types of hand factors: *DH* and *NDH*, as well as four types of speed factors: *S1*, 0.250 π rad/s; *S2*, 0.500 π rad/s; *S3*, 1.000 π rad/s; *S4*: 1.500 π rad/s. Through examinations of each parameter, hand and speed-dependent motor characteristics are to be studied. To be specific, the control mechanism of how humans respond to an instantaneous velocity input is to be investigated. Through *IPV2*, a magnitude-related property in velocity-control process and accuracy of velocity prediction can be examined while *IPT2* and *TD2* directly propose temporal behaviors of velocity-control performance such as reactivity or sensitivity to a given velocity input.

Performances of the 29 subjects were analyzed using MATLAB 2019a, MathWorks (Natick, Massachusetts, United States). A Butterworth IIR digital filter (3rd-ordered low-pass with a cutoff frequency of 7.5 Hz) was applied to all the data prior to analysis. SPSS statistics V26 (IBM) was used to carry out two-way repeated-measures analysis of variance (ANOVA) tests to numerically analyze differences in performances of the circular tracking movements with the hand and speed factors. Mauchly's sphericity test (*p* > 0.05) was also executed to validate the results of ANOVA. For the results whose sphericity was not assumed (*p* < 0.05), the values corrected with Greenhouse–Geisser in tests of within-subjects effects were employed. The post hoc test was performed by Bonferroni pairwise comparison (*p* < 0.05) to verify the significance of each condition for hand and speed factors.

#### **3. Results**

The designed experiment was to investigate initial motor characteristics in the target tracking movements. To examine speed-dependent characteristics of the transient responses and the differences between the *DH* and *NDH*, 29 subjects participated in the experiment using their *DH* and *NDH* at the four speeds. The obtained tracer's position data was transformed into three parameters: *IPV2*, *IPT2*, and *TD2*.

Figure 2 shows typical examples of the calculated velocities of target-tracking movements at *S2*, while the target rotated through the first half-orbit of the circular trajectory. The measured trajectories of the *DH* and *NDH* are illustrated on the frontal plane (Figure 2A,B). The velocity components of the hands are depicted on the *X*-axis (Figure 2C1,D1) and *Y*-axis (Figure 2C2,D2). The normalized speeds are displayed in Figure 2C3,D3 in terms of the target speed *S2*; the computed *IPV2*, *IPT2*, and *TD2* are marked.

**Figure 2.** Typical examples of velocities during the first half-orbit of circular tracking movements at 0.500 π rad/s (S2). (**A**) Trajectories of the target and tracer performed by the dominant hand (*DH*) on the frontal plane. (**B**) Trajectories of the target and tracer performed by the non-dominant hand (*NDH*) on the frontal plane. (C1) Velocity of the *DH* calculated on the *X*-axis and (C2) on the *Y*-axis. (C3) Computed speeds on the frontal plane, normalized to *S2*. (D1) Velocity of the *NDH* calculated on the *X*-axis and (D2) on the *Y*-axis. (D3) Computed speeds on the frontal plane, normalized to *S2*.

### *3.1. Di*ff*erences in the Transient Response of Tracking Movement Based on IPV2*

ANOVA of the *IPV2* demonstrated that only the speed factor caused a significant difference in the *IPV2* of the transient responses of the circular tracking movement (item A in Table S1). The hand factor and the interaction between the hand and speed factors did not significantly affect the *IPV2*.

Figure 3 illustrates the *IPV2* results of the *DH* and *NDH* in terms of the transient responses of the tracking movement on the 2D frontal plane. The statistical values of *IPV2* are listed as the variable *IPV2* in Table 2. There was no significant difference in *IPV2* between the *DH* and *NDH* (Figure 3A,B; item B in Table S1). The *IPV2* of the *DH* indicated that the maximum speed differences between the target and the tracer were approximately 37.70% and 26.22% larger at *S1* and *S2*, respectively, than at *S4* (Figure 3C; item C in Table S1). The *IPV2* of the *NDH* indicated that the maximum speed differences between the target and the tracer were approximately 55.05% and 35.30% larger, respectively, at *S1* and *S2* than at *S3* as well as 62.79% and 43.04% larger at *S1* and *S2* than at *S4* (Figure 3D; item D in

Table S1). The combined *IPV2* suggested that the maximum speed differences between the target and the tracer were approximately 39.25% and 23.63% larger at *S1* and *S2* than at *S3* as well as 50.25% and 34.63% larger at *S1* and *S2* than at *S4* (Figure 3E; item E in Table S1).

**Figure 3.** Evaluation of transient responses of the circular tracking movement based on initial peak velocity (*IPV2)* on the frontal plane. (**A**) The values of *IPV2* of 29 subjects for four speed factor levels. (**B**) The pairwise comparison between the dominant hand (*DH*) and *NDH* was shown for *IPV2* of all the speed factors. (**C**) Pairwise comparisons of *IPV2* were represented for the speed factor within *DH* and (**D**) within *NDH*. (**E**) The results of pairwise comparison of *IPV2* in both *DH* and *NDH*.

\* These values are recorded in terms of 'mean (± standard deviation)'.

These results signified that *IPV2* was likely to decrease as the target speed became faster on the 2D frontal plane regardless of the hand. In other words, the subjects' hands showed clear tendencies to reduce the initial speed difference by approximately 16.75% at a faster speed.

### *3.2. Di*ff*erences in Transient Response of Tracking Movement Based on IPT2*

ANOVA of the *IPT2* revealed that only the hand factor caused significant differences in the *IPT2* of the transient responses of the circular tracking movement (item A in Table S2). The speed factor and the interaction between the hand and speed factors did not significantly affect the *IPT2*.

Figure 4 illustrates the *IPT2* results of the *DH* and *NDH* in terms of the transient responses of the tracking movement on the 2D frontal plane. The statistical values of *IPT2* are listed as variable *IPT2* in Table 2. Although no significant differences were found in the paired *IPT2* (Figure 4A; item B in Table S2), the *IPT2* of the hands revealed that the *DH* reached the initial peak speed approximately 0.0423 s before the *NDH* did, regardless of the target speed (Figure 4B). The *IPT2* values of the *DH* and *NDH* indicated that there were no significant differences in the speed factor (Figure 4C–E; items C–E in Table S2).

**Figure 4.** Evaluation of the transient responses of the circular tracking movement based on the initial peak time (*IPT2)* on the frontal plane. (**A**) The values of *IPT2* of the 29 subjects at four speed factor levels. (**B**) The pairwise comparison between dominant hand (*DH*) and non-dominant hand (*NDH*) is shown for the *IPT2* of all the speed factors. (**C**) Pairwise comparisons of the *IPT2* are represented in terms of the speed factor within *DH* and (**D**) *NDH*. (**E**) Pairwise comparison of *IPT2* for both *DH* and *NDH*.

The stated results implied that the *IPT2* had a significant relationship with the hand factor on the 2D frontal plane. Specifically, the subjects' *DH* attained its initial peak velocity 0.0423 s before the *NDH* did at the beginning of the tracking movement while both the hands disclosed no clear tendency on the target speed.

### *3.3. Di*ff*erences in Transient Response of Tracking Movement Based on TD2*

ANOVA of the *TD2* indicated that all the factors and their interaction caused significant differences in the *TD2* of the transient responses of the circular tracking movement (item A in Table S3).

Figure 5 illustrates the *TD2* results of the *DH* and *NDH* during the transient responses of the tracking movement on the 2D frontal plane. The statistical values of *TD2* are listed as variable *TD2* in Table 2. Paired analysis of *TD2* implied that the subjects initiated tracking approximately 0.0494 s earlier at *S1* when using the *DH* than when using the *NDH*; at *S2*, they initiated tracking 0.0709 s earlier (Figure 5A; item B in Table S3). The *TD2* of the hands revealed that the *DH* started moving the tracer approximately 0.0417 s earlier than the *NDH*, irrespective of the target speed (Figure 5B).

**Figure 5.** Evaluation of transient responses of the circular tracking movement based on the time delay (TD2) on the frontal plane. (**A**) The values of TD2 of 29 subjects at four speed factor levels. (**B**) The pairwise comparison between dominant hand (DH) and non-dominant hand (NDH) is shown for the TD2 of all speed factors. (**C**) Pairwise comparisons of TD2 are represented for each speed factor with the DH and (**D**) NDH. (**E**) Pairwise comparison of TD2 with both the DH and NDH.

The *TD2* of the *DH* indicated that the subjects initiated tracking approximately 0.0406 s earlier at *S2* than at *S1*, and 0.0567 s earlier at *S4* than at *S1* (Figure 5C; item C in Table S3), whereas the *TD2* of the *NDH* indicated that the subjects reacted to the target approximately 0.0716 and 0.0743 s earlier at *S3* and *S4* than at *S1*, and that they reacted 0.0525 and 0.0552 s earlier at *S3* and *S4* than at *S2* (Figure 5D; item D in Table S3). The combined *TD2* suggested that the subjects started moving the tracer approximately 0.0299, 0.0544, and 0.0655 s earlier at *S2*, *S3*, and *S4*, respectively, than at *S1*, as well as 0.0356 s earlier at *S2* than at *S4* (Figure 5E; item E in Table S3).

These results signified that *TD2* showed significant differences between slow (*S1*, *S2*) and fast (*S3*, *S4*) speeds. In other words, the subjects reacted to the target approximately 0.0218 s more quickly at fast target speeds in the circular tracking movement. Moreover, the *TD2* of the *DH* and *NDH* showed different statistical features; specifically, the subjects had a transient response that was approximately 0.0417 s earlier with the *DH* than with the *NDH*.

### **4. Discussion**

This study quantitatively analyzed the transient responses of the circular tracking movement of the *DH* and *NDH* to investigate initial motor control characteristics at different target speeds. Three parameters were examined: *IPV2*, the normalized magnitude of initial peak velocity, *IPT2*, initial peak velocity time, and *TD2*, the reaction time the subject took to react to the target. All parameters were investigated at *S1*–*S4* with the *DH* and *NDH*.

The following discussion mainly focuses on: (1) speed-dependent features of *IPV2*, (2) temporal analysis of *IPT2* and *TD2*, and (3) clinical applications and future work.

#### *4.1. Speed-Dependent Features of IPV2*

The subjects performed the target-tracking movement by immediately moving the tracer faster than the target after they perceived target movement. As overshoot is a key component of transient response analysis in control theories [54], the speed of the tracer exceeded that of the target and, after the target was caught up, the subjects continued the tracking task by adjusting the speed of the tracer (Figure 2). Particularly at *S4*, the subjects controlled the speed of the tracer rather than reducing the position error between the target and the tracer. This implied that they deemed the target too fast to track and that they simply decided to maintain a certain level of the tracer's speed.

Analyzing the immediate movements of the tracer when the target started moving, we showed that the *IPV2* decreased as the target speed increased, indicating that the initial prediction about the target speed was more accurate at faster speeds. The subjects could barely apprehend the next target speed because the given instruction was not quantitative, so the initial target-tracking movement was executed based solely on the subject's predicted speed. The *DH* and *NDH* commonly had lower *IPV2* values at faster speeds and showed nearly identical behavior in overall speed, indicating that both hands have the same internal model for initiating tracking movement.

### *4.2. Temporal Analysis of IPT2 and TD2*

The *IPT2* represents the time corresponding to *IPV2* and therefore indicates the moment when the subject first corrected the speed of the tracer toward the moving target. In other words, *IPT2* expresses the actual initiation of speed control. The *IPT2* showed no significant difference among the different speeds, but had a 0.0423-s difference between the *DH* and *NDH*. It follows that both hands may have nearly the same reactivity to the target speed. This also suggests that an identical internal model controls motor performances of the hands. Such an idea coincides with motor equivalence, the theoretical concept that the central nervous system (CNS) rather stores abstract forms of information on actions than records neural commands to specific muscles for performing actions [55]. The possibility for the hands to receive not distinct commands from the CNS but similar forms of information intimates that the *IPT2* could be a personal motor characteristic rather than an externally affected trait.

*TD2*, the time when the subject perceived the target's positional change and initiated tracer movement, showed significant differences between the hands and among the various speeds. In the *DH* and *NDH*, the *TD2* reduced as the target became faster, showing clear differences in average values. Specifically, mutual differences were found between the slow (*S1*, *S2*) and fast (*S3*, *S4*) speeds, indicating that when the subjects performed more trials, they predicted the starting moment of trials more accurately. In addition, the *DH* had less *TD2* than the *NDH* at all the speeds, showing that the subjects were more proficient at controlling the *DH* than the *NDH*. Otherwise stated, the *DH*, comparatively the more experienced hand, reacted to the target faster than the *NDH*. Assuming that the same internal model controlled the movements of the *DH* and *NDH*, the tracers might have started moving at a nearly identical moment in each case. Since the *TD2* showed a significant difference between the hands, we concluded that the *DH* showed better performance at controlling objects in reacting to motor commands and catching the moving target.

### *4.3. Clinical Application and Future Work*

Assessing of motor functions and analysis of possible recovery processes will be feasible by applying the protocol of this study to patients with restrictions on upper limb movements. The experimental system can measure performance of a circular target tracking movement and can evaluate motor characteristics of an upper limb quantitatively. The examined results can be integrated with conventional indicators which inspect motor capability of individual joint movements and diagnose current recovery processes from deficits of a neurological disorder, such as Fugl-Meyer

assessment (FMA) and manual function test (MFT). After appropriate correlation analysis, the present protocol will be employed in estimating the patients' physical conditions, for example, spasticity of their impaired limbs, and in speculating potential recovery phases.

The results of this research can benefit the clinical population with neural impairment by providing valuable information which can be employed in navigating suitable treatment. Such patients have difficulties in performing continuous multi-joint movements for a long time because of sensorimotor malfunction caused by the impairment. Motor characteristics of the patients can be examined by simply conducting a half orbit of the circular tracking movement. The obtained information can be effectively useful for therapists in planning proper rehabilitation goals or in designing customized treatment programs for the patients. The outcome of this research is expected to provide clinical improvement to those whose motor functions are deteriorated by a stroke, spatial neglect, or lesion in the cerebellum.

In future, we will investigate target-tracking performance at random target speeds without notifying the subjects about the target speed. Without any clues about the target speed, it will likely be difficult for subjects to make any predictions. The motor control characteristics of individuals will be studied by analyzing reactivity to a moving target.

*TD2* represented time delay or reaction time of the target tracking movement in this research. In neuroscience and biological engineering, reaction time is defined as the time taken when a neural command from the brain is transmitted to its musculoskeletal system for a joint movement, and the time is estimated as approximately 200 msec [45,46]. A time delay in control theories is normally referred to as a time necessary to reach 50% of the final value in a step response of a controlled system [54]. Both definitions were difficult to apply in this research since the measured data did not clearly indicate the starting moment of the reaction and final values of the subjects' angular velocities in their tracking movement. We will evaluate these definitions by executing the same tasks within a dynamically-limited environment in future work.

A mathematical model to explain the tradeoff characteristic between speed and accuracy could be developed in the future. Our previous studies verified that the subjects could stably perform the tracking tasks after the first orbit of the trajectory [37,38]. In terms of control strategies, the subjects employed relatively more intermittent feedback control at the slow target speeds whereas feedforward control was dominant at the fast target speeds. This tradeoff relationship has been interpreted between speed and accuracy by establishing computational models [56,57]. Applying those mathematical models in our results is anticipated to better explain the speed-dependent characteristics of the human motor control mechanism.

Furthermore, an identical experiment with different weights of the hand-held controller will be conducted to investigate the effects of weight on the circular tracking movement. Applying a constant weight to the gravitational direction will probably influence the transient response of the tracking movement. Such an experiment should elucidate any differences in the impedance characteristics of *DH* and *NDH*, and analyze the motor characteristics of the hands from various perspectives.

### **5. Conclusions**

The initial motor control characteristics of *DH* and *NDH* in the circular tracking movement were examined in a 3D VR space. The transient responses of the hands in the tracking movement were analyzed for *S1*–*S4* on the frontal plane. The *IPV2* decreased as the target speed increased, irrespective of the hand, suggesting that the subjects improved velocity-prediction accuracy by approximately 16.75% at faster speeds. The *IPT2* did not vary with the speed factor but showed a 0.0423-s difference between the *DH* and *NDH*, indicating that the *DH* reached its maximum speed earlier than the *NDH*. The *TD2* revealed significant differences in both the hand and speed factors, implying that the subjects reacted to the target 0.0218 s earlier as the target speed increased, and that the *DH* reacted to the target 0.0417 s earlier than the *NDH*. The performances of the *IPV2* and *IPT2* suggested that the same internal model controls tracking movements of both the hands whereas the results of *TD2* proposed that the *DH* generally had better reactivity in its transient response. The target tracking movements can be

utilized in clinic environments to evaluate motor functions of patients and to set up treatment plans for rehabilitation. Effects of different experimental factors such as random speeds or weights will be analyzed as well as applying different methods of calculating reaction time and mathematical models of the tradeoff characteristics to the target tracking movements. This study examined the motor control characteristics of the *DH* and *NDH* in their transient responses of the circular tracking movements in the 3D VR space.

**Supplementary Materials:** The following are available online at http://www.mdpi.com/1424-8220/20/12/3477/s1, Table S1: a summary of statistical analysis of *IPV2* for the circular tracking movement, Table S2: a summary of statistical analysis of *IPT2* for the circular tracking movement, Table S3: a summary of statistical analysis of *TD2* for the circular tracking movement.

**Author Contributions:** Conceptualization, W.P., J.K., and W.C.; methodology, W.P. and W.C.; resources, J.K.; software, W.C.; data curation, W.P. and W.C.; investigation, W.P. and W.C.; validation, W.P., W.C., formal analysis, W.P., W.C., H.J., and G.L.; writing—original draft, W.P., H.J., and G.L.; writing—review and editing, W.P., W.C., and J.K.; All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the Korea National Research Foundation (KNRF) grant funded by the Korean government (MOE) (No. 2020R1I1A3A04038203).

**Conflicts of Interest:** The authors declare no conflicts of interest.

### **References**

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## **Turning Characteristics of the More-A**ff**ected Side in Parkinson's Disease Patients with Freezing of Gait**

**Hwayoung Park 1, Changhong Youm 1,2,\*, Myeounggon Lee 1, Byungjoo Noh 2 and Sang-Myung Cheon 3**

Received: 8 May 2020; Accepted: 29 May 2020; Published: 30 May 2020

**Abstract:** This study investigated the turning characteristics of the more-affected limbs in Parkinson's disease (PD) patients in comparison with that of a control group, and in PD patients with freezing of gait (FOG; freezers) in comparison with those without FOG (non-freezers) for 360◦ and 540◦ turning tasks at the maximum speed. A total of 12 freezers, 12 non-freezers, and 12 controls participated in this study. The PD patients showed significantly longer total durations, shorter inner and outer step lengths, and greater anterior–posterior (AP) root mean square (RMS) center of mass (COM) distances compared to those for the controls. The freezers showed significantly greater AP and medial-lateral (ML) RMS COM distances compared to those of non-freezers. The turning task toward the inner step of the more-affected side (IMA) in PD patients showed significantly greater step width, total steps, and AP and ML RMS COM distances than that toward the outer step of the more-affected side (OMA). The corresponding results for freezers revealed significantly higher total steps and shorter inner step length during the 540◦ turn toward the IMA than that toward the OMA. Therefore, PD patients and freezers exhibited greater turning difficulty in performing challenging turning tasks such as turning with an increased angle and speed and toward the more-affected side.

**Keywords:** Parkinson's disease; freezing of gait; gait; turning; fall; kinematics

### **1. Introduction**

Parkinson's disease (PD) is one of the most common neurodegenerative disorders [1]. A total of 60%–80% of patients with PD eventually develop the freezing of gait (FOG) [1], which is defined as the "episodic absence or marked reduction in forward progression of the feet despite the intention to walk" [2]. The FOG is most commonly experienced during turning tasks, gait initiation, walking through narrow passages, timed up and go (TUG) tasks, dual tasks, and while approaching a destination or avoiding an obstacle [3]. In particular, PD patients with FOG (freezers) encounter greater challenges and need to pay greater attention when performing turning tasks [4]. They face increased risk of falls caused by the instability of their body because it requires the center of mass (COM) to momentarily shift outside the lateral boundaries of the base of support [5].

Previous studies have reported the turning characteristics of 0◦, 90◦, 120◦, and 180◦ turns [6]; 180◦ and 360◦ turns while walking [7]; and 180◦ turns while walking and 360◦ turns in place with a narrower range [1] in PD patients and freezers. These studies have revealed that freezers encounter greater difficulty in performing turning tasks than non-freezers, as evidenced by the higher number of steps, slower turning speed, shorter step length, and wider step width with an increasing turning

angle. In addition, freezers impose greater demand on lower limb coordination and experience an increased risk of falling and frequent freezing episodes in daily life [6,7].

More than 50% of PD patients exhibit motor impairments, in which unilateral predominance may persist throughout the progression of the disease [8,9]. However, most patients present bilateral impairments at a later stage of the disease [10]. One reason for the unilateral predominance of disease symptoms appears to be the differences in the striatal uptake between the caudate and putamen nuclei [11]. Additionally, the dominant cerebral hemisphere extensively distributed throughout the circuitry of basal ganglia [12]. The asymmetrical gait characteristics of PD patients affect both the upper and lower limbs, and they appear to affect balance control during turning tasks [13]. However, although PD patients show unilateral predominance of disease symptoms, turning tasks assigned in studies so far have been used to evaluate the preferred turning direction toward one side or both sides, without considering the turning task in the direction of the more-affected side.

Spildooren et al. [14] reported that freezers showed higher cadence while turning toward the more-affected side compared to non-freezers. Their findings suggested that freezers experienced greater turning difficulty when turning toward the more-affected side. However, it is still unclear whether there is a difference in the turning characteristics while turning toward the more-affected side in PD patients and freezers. Although PD patients and freezers are affected by the turning speed [15,16], there has been limited research on the characteristics of turning tasks performed at the maximum speed.

Several studies have shown that greater attention and postural stability are required when increasing the turning angle [1,16]. An experimental choice was made to select a 540◦ turning angle as the challenging condition that would threaten posture stability. This study investigated the characteristics of the turning direction toward the more-affected side in PD patients in comparison with controls, and in freezers in comparison with non-freezers for 360◦ and 540◦ turning tasks at the maximum speed. We hypothesized that, for the turning direction of the more-affected side, the turning characteristics with regard to the turning direction may significantly differ between PD patients and controls, and freezers and non-freezers.

### **2. Materials and Methods**

### *2.1. Participants*

A total of 60 PD patients were recruited. However, 36 of them could not be considered because of reasons such as the withdrawal of informed consent, refusal to participate in the experiment, or inability to participate in the experiment. Eventually, 24 PD patients, which included 12 freezers and 12 non-freezers, were considered, and 12 age-matched older adults also participated as controls. The consort diagram containing the details of the study participants is shown in Figure 1 and Table 1. Idiopathic PD patients were diagnosed by a neurologist using the UK Parkinson's Disease Society Brain Bank criteria [17]. The inclusion criteria were: (a) Age 50–75 years; (b) capable of walking and moving independently and with a modified Hoehn and Yahr (H&Y) stage of 2–3 [18,19]; (c) mini-mental state examination (MMSE) score of >24 [20]; (d) stable response to antiparkinsonian medications; and (e) classified as freezers with a score of >3 according to the new freezing of gait questionnaire (NFOGQ) [21].

None of the participants (PD patients and controls) reported any history of musculoskeletal injuries or other cardiovascular and neurological diseases in the past six months. The PD patients that required movement assistive devices and those with dyskinesia (uncontrollable muscle movements induced by drug therapy) were excluded. All participants provided written informed consent before participating in this study. The experimental protocols were approved by the institutional review board (IRB) of Dong-A University Medical Center (IRB number: DAUHIRB-17-033 (See Supplementary file S1. IRB number)).

**Figure 1.** Consort flow diagram. NFOGQ: New freezing of gait questionnaire.

**Table 1.** Physical characteristics of all participants.

All data represent the mean ± standard deviation; PD: Parkinson's disease; BMI: Body mass index; MMSE: Mini- mental state examination; UPDRS: Unified Parkinson's disease rating scale; L-dopa: Levodopa. Boldface denotes a significant difference between freezers and non-freezers. Significant difference: *p* < 0.05.

### *2.2. Test Procedures*

The PD patients were in the "On" state, which was induced by dopaminergic medication that was administered approximately 2–3 h before the tests. They were sufficiently under the effect of the medication [22]. The experiments were performed in two sessions. In the first session, the participants completed the informed consent form and were assessed according to the Unified Parkinson disease rating scale (UPDRS) [23], modified H&Y stage, NFOGQ, and MMSE (Table 1). In the second session, the participants warmed up for approximately 5 min before performing the turning tasks and practiced such tasks. After 5 min of rest, the test started, and the participants were instructed to successfully complete the 360◦ and 540◦ turning tasks at the maximum speed three times.

The orientation of body segments in a global coordinate system was captured using nine cameras (Vicon MX-T10, Oxford Metrics, Oxford, UK). The data sampling frequency was 100 Hz [24]. Three-dimensional motion data were captured using the Vicon Nexus software (version 1.83; Oxford Metrics, UK). The appropriate metrics were measured bilaterally to estimate the joint kinematic data. The plug-in gait marker set was attached with 39 14-mm spherical reflective markers, which is a modified version of the Helen Hayes marker set [25].

The 360◦ and 540◦ turning tasks were conducted with maximum speed, which was defined as the fastest speed at which the patients choose to perform the tasks, in both the directions of the inner step of the more-affected side (IMA) and the outer step of the more-affected side (OMA). The turning directions for the more-affected sides (IMA and OMA) were classified by a neurologist. The 360◦ and 540◦ turning tasks with maximum speed used in this study were modified from the TUG test (Figure 2).

**Figure 2.** Schematic of the data collection and analysis phase. (**a**) The 360◦ turning task involved sitting on the chair, getting up from the chair, turning around the cone at the 3-m point, and reaching the final target point as quickly as possible. The 540◦ turning task involved sitting on the chair, getting up from the chair, turning around the cone at the 3-m point, and reaching the chair as quickly as possible. (**b**,**c**) In the analysis phase of the 360◦ and 540◦ turning tasks, the start event was defined as the event when the angle between the pelvic vector and the left and right vectors passed 10◦. After the turning movement was completed, the event when the two vectors completed 360◦ and 540◦ was defined as the end of the rotation. (**d**) Step width was defined as the length between the initial foot heel contact of one leg and the initial foot heel contact of the other leg. The inner and outer step lengths were defined as the lengths of the initial foot heel contact of the left/right leg and the initial foot heel contact of the other leg, respectively.

### *2.3. Data Analysis*

The 360◦ and 540◦ turning data was a fourth-order Butterworth low-pass filtered with a cutoff frequency of 10 Hz [24]. The measurements were performed three times and the averaged value was used for the analysis.

The variables for the analysis were the total steps, total duration, step width, inner and outer step lengths, area of the 95% confidence interval (CI), and the anterior–posterior (AP) and medial–lateral (ML) root mean square (RMS) distances of the center of mass (COM) during the 360◦ and 540◦ turning tasks performed at maximum speed. A weighted sum of the COM of all segments was calculated, where the segments were defined by markers. The COM was the COM of all modeled segments. In addition, the area of the 95% CI (Equation (1)) and the AP and ML RMS distances of the COM on the horizontal plane (Equation (2)) were computed as [26]:

$$\text{Area} = \pi \text{ab},\tag{1}$$

where a and b denote the major and minor axis, respectively.

$$\mathbf{a} = \left[ 3\left( \text{SD}\_{\text{AP}}^2 + \text{SD}\_{\text{ML}}^2 + \text{D} \right) \right]^{\frac{1}{2}}; \mathbf{b} = \left[ 3\left( \text{SD}\_{\text{AP}}^2 + \text{SD}\_{\text{ML}}^2 - \text{D} \right) \right]^{\frac{1}{2}},\tag{2}$$

where SDAP and SDML denote the standard deviations of the AP and ML RMS distances of the COM, respectively.

$$\text{PD} = \left[ \left( \text{SD}\_{\text{AP}}^2 + \text{SD}\_{\text{ML}}^2 \right) - 4 \left( \text{SD}\_{\text{AP}}^2 \text{SD}\_{\text{ML}}^2 - \text{SD}\_{\text{APML}}^2 \right)^{\frac{1}{2}} \right. \tag{3}$$

where D denotes the distance from the center to the focus for the AP and ML RMS distances of the COM.

$$\text{SD}\_{\text{APML}} = \frac{1}{\text{N}} \left( \sum\_{n=1}^{\text{N}} \chi\_{\text{AP}(n)} \chi\_{\text{ML}(n)} \right) \tag{4}$$

where XAP and XML denote the position of the AP and ML of the COM, respectively.

$$\text{RMS}\_{\text{AP}} = \left[ \frac{1}{\text{N}} \sum\_{\mathbf{n}=1}^{\text{N}} \left( \chi\_{\text{AP}(\mathbf{n})} - \overline{\chi}\_{\text{AP}} \right)^2 \right]^{\frac{1}{2}}; \text{RMS}\_{\text{ML}} = \left[ \frac{1}{\text{N}} \sum\_{\mathbf{n}=1}^{\text{N}} \left( \chi\_{\text{ML}(\mathbf{n})} - \overline{\chi}\_{\text{ML}} \right)^2 \right]^{\frac{1}{2}},\tag{5}$$

where XAP and XML denote the mean of AP and ML of the COM, respectively.

#### *2.4. Statistical Analysis*

The Shapiro–Wilk test revealed that data followed a normal distribution. A two-way analysis of variance (ANOVA) with repeated measures was performed to analyze the differences between the groups (PD patients compared with controls and freezers compared with non-freezers) and between the turning directions (IMA and OMA) during the 360◦ and 540◦ turning tasks at maximum speed. *Post-hoc* tests were performed using the independent sample t-test between the groups and the paired samples t-test was used to analyze the turning directions. To compare the differences of physical characteristics between all participants, one-way ANOVAs were performed.

In addition, the responsiveness between the groups and the turning directions were expressed as the effect size (ES). Univariate and multivariate logistic regression analyses and stepwise binary logistic regression analysis were performed to identify the classifier variables for PD patients and freezers, and for the turning directions. Classifier variables were expressed as odds ratios (OR) with a 95% CI (min to max). All statistical analyses were performed using SPSS 22.0 (SPSS Inc., Chicago, IL, USA), and the statistical significance was set at 0.05.

### **3. Results**

### *3.1. Di*ff*erences between PD Patients and Controls*

In the results of the 360◦ turning task at the maximum speed, the total steps (IMA, *p* = 0.035), total duration (IMA, *p* = 0.006; OMA, *p* = 0.047), inner (IMA, *p* = 0.003; OMA, *p* < 0.001) and outer (IMA, *p* = 0.001; OMA, *p* < 0.001) step lengths, and ML RMS distance of the COM (IMA, *p* = 0.042; OMA, *p* = 0.016) of the turning tasks in the directions of the IMA and OMA showed significant differences between PD patients and controls (Table 2). In the results of the 540◦ turning task at the maximum speed, the inner (OMA, *p* = 0.018) and outer (IMA, *p* = 0.001) step lengths and AP RMS

distance of the COM (IMA, *p* = 0.042; OMA, *p* = 0.016) of the turning tasks in the directions of the IMA and OMA showed significant differences between the PD patients and controls (Table 3).

**Table 2.** Comparison of turning characteristics of PD patients and controls for the 360◦ turning task at maximum speed.

All data represent the mean ± standard deviation (95% confidence interval). *pa*-value (groups): Independent t-tests; *pb*-value (turning direction): Paired samples t-tests; ES: Effect size; IMA: Inner step of the more affected side; OMA: Outer step of the more affected side; COM: Center of mass; RMS: Root mean square. Boldface indicates a significant difference (*p* < 0.05).

### *3.2. Classifier Variables for PD Patients and Controls*

Stepwise binary logistic regression analysis for the PD patients and controls revealed that the total duration (OR: 0.271; 95% CI: 0.077–0.957; RN2 = 0.532; *p* = 0.043) and outer step length (OR: 4.826; 95% CI: 1.248–18.659; RN2 = 0.532; *p* = 0.023) of the turning task performed in the direction of the IMA and inner step length (OR: 7.218; 95% CI: 1.831–28.463; RN2 = 0.607; *p* = 0.005) of the turning task performed in the direction of the OMA were significantly different during the 360◦ turning task at the maximum speed. In addition, the outer step length (OR: 5.642; 95% CI: 1.578–20.167; RN2 = 0.390; *p* = 0.008) of the turning task performed in the direction of the IMA and inner step length (OR: 2.984; 95% CI: 1.092–8.152; RN2 = 0.223; *p* = 0.033) of the turning task performed in the direction of the OMA were significantly different during the 540◦ turning task at the maximum speed.

**Table 3.** Comparison of turning characteristics of PD patients and controls for the 540◦ turning task at maximum speed.

All data represent the mean ± standard deviation (95% confidence interval). *pa*-value (groups): Independent t-tests; *pb*-value (turning direction): Paired samples t-tests. Boldface indicates a significant difference (*p* < 0.05).

### *3.3. Di*ff*erences between Freezers and Non-Freezers*

The AP (360◦ turn, *p* < 0.001; 540◦ turn, *p* < 0.001) and ML (360◦ turn, *p* < 0.001; 540◦ turn, *p* < 0.001) RMS distances of the COM in the turning task performed in the direction of the IMA (Figure 3) and the ML RMS distance of the COM (360◦ turn, *p* = 0.003; 540◦ turn, *p* = 0.013) in the turning task performed in the direction of the OMA were significantly different for freezers and non-freezers during the 360◦ and 540◦ turning tasks at the maximum speed (Tables 4 and 5).

### *3.4. Classifier Variables for Freezers and Non-Freezers*

Stepwise binary logistic regression analysis for freezers and non-freezers showed that the ML RMS distance of the COM in the turning tasks performed in the directions of the IMA (OR: 10.188; 95% CI: 1.699–61.100; RN2 = 0.596; *p* = 0.011) and OMA (OR: 0.193; 95% CI: 0.053–0.708; RN2 = 0.437; *p* = 0.013) were significantly different during the 360◦ turning task at the maximum speed. In addition, the inner step length (OR: 0.047; 95% CI: 0.004–0.560; RN2 = 0.665; *p* = 0.016) of the turning task performed in the direction of the IMA and the ML RMS distance of the COM (OR: 0.276; 95% CI: 0.086–0.883; RN2 = 0.324; *p* = 0.030) in the turning task performed in the direction of the OMA were significantly different during the 540◦ turning task at the maximum speed.

**Figure 3.** Single participant trial of freezers, non-freezers, and controls during the 360◦ and 540◦ turning tasks. This figure presented the horizontal trajectory of the center of mass presented by large (a freezer) and small (a non-freezer) solid dots, and full line (a control), respectively.

All data represent the mean ± standard deviation (95% confidence interval). *pa*-value (groups): Independent t-tests; *pb*-value (turning direction): Paired samples t-tests. Boldface indicates a significant difference (*p* < 0.05).

**Table 5.** Comparison of turning characteristics of freezers and non-freezers for the 540◦ turning task at maximum speed.

All data represent the mean ± standard deviation (95% confidence interval). *pa*-value (groups): Independent t-tests; *pb*-value (turning direction): Paired samples t-tests. Boldface indicates a significant difference (*p* < 0.05).

### *3.5. Di*ff*erences between Turning Characteristics in the Directions of the IMA and OMA*

In the results of the PD patients, the step width (*p* = 0.042) and AP (*p* = 0.009) and ML (*p* = 0.002) RMS distances of the COM in the turning tasks performed in the direction of the IMA were significantly higher than those in the direction of the OMA during the 360◦ turning task at the maximum speed (Table 2). In addition, the total steps (*p* = 0.008) and AP (*p* = 0.005) and ML (*p* = 0.003) RMS distances of the COM in the turning tasks performed in the direction of the IMA were significantly higher than those in the direction of the OMA during the 540◦ turning task at the maximum speed (Table 3). In the results of the freezers, the total steps (*p* = 0.026) and inner step length (*p* = 0.007) of the turning tasks performed in the direction of the IMA were significantly higher than those in the direction of the OMA during the 540◦ turning task at the maximum speed (Table 5).

#### *3.6. Classifier Variables for Turning Tasks in the Directions of the IMA and OMA*

Stepwise binary logistic regression analysis for the PD patients revealed that the ML RMS distance of the COM was significantly different in the turning task performed in the direction of the IMA during the 360◦ (OR: 0.000; 95% CI: 0.000–0.005; RN2 = 0.301; *p* = 0.002) and 540◦ (OR: 0.000; 95% CI: 0.000–0.021; RN2 = 0.231; *p* = 0.004) turning tasks at the maximum speed. In addition, the results for freezers revealed that the ML RMS distance of the COM was significantly different in the turning direction of the IMA during the 360◦ (OR: 0.000; 95% CI: 0.000–0.010; RN2 = 0.405; *p* = 0.005) and 540◦ (OR: 0.000; 95% CI: 0.000–0.014; RN2 = 0.426; *p* = 0.004) turning tasks at the maximum speed. Significant differences were found for the total steps (OR: 0.755; 95% CI: 0.574–0.993; RN2 = 0.426; *p* = 0.045) in the 540◦ turning task performed in the direction of the IMA at the maximum speed.

### **4. Discussion**

### *4.1. Di*ff*erences between PD Patients and Controls, and Freezers and Non-Freezers*

We found that PD patients showed a significantly longer total duration, shorter inner and outer step lengths, and greater ML RMS distance of the COM during the 360◦ turning task at the maximum speed compared to those of the controls. The total duration and outer step length of the 360◦ and 540◦ turning tasks performed in the direction of the IMA and the inner step length of the 360◦ and 540◦ turning tasks performed in the direction of the OMA at the maximum speed were revealed as the classifier variables for the PD patients. In addition, the freezers showed significantly greater AP and ML RMS distances of the COM in the 360◦ and 540◦ turning tasks performed in the direction of the IMA, and greater ML RMS distance of the COM in the 360◦ and 540◦ turning tasks performed in the direction of the OMA at the maximum speed in comparison with those of the non-freezers. The ML RMS distance of the COM in the 360◦ and 540◦ turning tasks performed in the directions of the IMA and OMA at the maximum speed were revealed as the classifier variables for the freezers. These results demonstrated our hypothesis that turning tasks under challenging conditions might be more difficult for PD patients than for controls, and for freezers than for non-freezers.

Turning tasks threaten the stability of PD patients more than any other freezing trigger, and they require the precise control of each limb [26]. Therefore, PD patients may experience greater challenge and distraction in performing turning tasks. Previous studies have reported that PD patients showed significantly greater total steps, wider step width, shorter step length, and slower turning speed to maintain their center of gravity between the two feet [27]. Increased turning angle and speed were observed during 90◦ and 180◦ turning tasks at the preferred speed [28], 180◦ turning tasks at comfortable and faster speeds [29], and 30◦ to 180◦ turning tasks at the self-selected preferred, slower, and faster speeds [15]. Thus, these turning characteristics may be a compensatory strategy for postural instability in PD patients [30].

The motor cortex closely cooperates with the basal ganglia, and the cerebellum is involved in some functions such as attention, executive and visuospatial functions, and control motor tasks [31]. The motor symptoms in PD patients are the results of incrementally impaired motor functions and impaired basal ganglia cells in the substantia nigra, which initiate and control body movements and balance [32]. Further, the degeneration of the substantia nigra is reflected as a loss of dopaminergic innervation, which may influence the control of automatized behavior. Therefore, PD patients may experience difficulty in automatic movements without attention [33]. The abnormalities in the basal ganglia or spinal cord circuit behavior have been suggested to be important factors influencing PD patients because of the greater decline in voluntary contractions [34]. The turning tasks thus require more attention and involve greater interlimb coordination, increased coupling between posture and gait [35], and modifications of locomotor patterns requiring frontal lobe cognitive and executive functions that control postural transitions [36]. Thus, these results suggest that challenging turning tasks may indicate postural instabilities, which imply that PD patients are more likely to be vulnerable to functional impairments [37] because turning is inherently asymmetrical and cannot stem from central pattern generators unlike walking in a straight line [38].

The results of our study show that PD patients might experience greater difficulty in 360◦ and 540◦ turning tasks in the directions of the IMA and OMA at the maximum speed in comparison with the controls, as evidenced by longer total turn durations, shorter inner and outer step lengths, and greater ML RMS distances of the COM (Figure 3). These differences were associated with relatively small effect sizes (d = 0.01–0.32). In addition, this study demonstrated classifier variables for the PD patients. We found that the total duration and the inner and outer step length may be considered when comparing PD patients and controls in addition to classifier variables to identify turning difficulty during the 360◦ and 540◦ turning tasks performed in the direction of the IMA and OMA at the maximum speed. Our results are useful for determining the decline in dynamic stability and the posture control of PD

patients, and they can potentially be used in clinical environments to measure the effectiveness of interventions meant to prevent risk of falls.

In previous studies, the freezers showed higher total steps, longer total durations, slower turning speeds, wider step widths, and shorter step lengths than the non-freezers, as well as increased turning angles and speeds while performing tasks such as 0◦, 90◦, 120◦, 180◦, and 360◦ turning tasks at the preferred and faster speeds [1,6,16]. Turning difficulty has been related to disorders of the frontocortical areas, which may be more severe in freezers than in non-freezers because freezers present a lower supplementary motor area activity than non-freezers [39]. The dysfunction of the supplementary motor area may require the functional reorganization of the spinal circuitry, given the abnormal projections from this area through the reticulospinal tract [40]. As the reticulospinal tract connects to the spinal interneurons that mediate one of the most powerful spinal inhibitory mechanisms, presynaptic inhibition [41], it is possible that presynaptic inhibition during challenging turning tasks may be deficient in freezers compared to those in controls and non-freezers. Therefore, increasing the turning angle and turning speed might pose additional difficulties in threatening situations, where dynamic stability while performing turning tasks with a small turning angle at the normal speed is potentially related to difficulties in coupling the inter-limb coordination and postural control [1,6]. In addition, such turning tasks in freezers may lead to an increased risk of fall because these conditions are not commonly experienced in daily life and therefore require greater attention and postural stability [15,29,42].

Further, our results show that freezers might experience greater difficulty in turning at 360◦ and 540◦ angles in the directions of the IMA and OMA at the maximum speed compared to non-freezers, as evidenced by the greater AP and ML RMS distances of the COM (Figure 3). These differences were associated with a relatively moderate effect size (d = 0.27–0.54). In addition, we found that the ML RMS distance of the COM may be considered when comparing freezers and non-freezers in addition to classifier variables to identify disease severity during the 360◦ and 540◦ turning tasks performed in the direction of the IMA and OMA at the maximum speed. Therefore, we suggest that preventive training to reduce risk of falling may require practicing around a larger radius to complete a turn [28,43].

#### *4.2. Di*ff*erences between Turning Characteristics in the Directions of the IMA and OMA*

We found that PD patients showed significantly greater step widths, higher total steps, and greater AP and ML RMS distances of the COM in the 360◦ and 540◦ turning tasks performed at the maximum speed in the direction of the IMA than those in the direction of the OMA. In addition, the ML RMS distance of the COM while turning by 360◦ and 540◦ at the maximum speed in the direction of the IMA was revealed as the classifier variable for the PD patients and controls. The 540◦ turning task performed by the freezers at the maximum speed revealed significantly higher total steps and shorter inner step length in the direction of the IMA than those in the direction of the OMA. Moreover, the ML RMS distance of the COM during the 360◦ turning task in the direction of the IMA and the total steps and ML RMS distance of the COM during the 540◦ turning task in the direction of the IMA at the maximum speed were revealed as the classifier variables for freezers and non-freezers. The study results proved our hypothesis that turning in the direction of the more-affected side might present more difficulties in PD patients and freezers.

Only one study on turning tasks toward the more-affected and less-affected sides in PD patients and freezers [14] reported that such patients showed higher turning steps and a greater duration of turn while turning 180◦ and 360◦ toward the more-affected side. Thus, PD patients and freezers might experience greater difficulty in turning toward the more-affected side [14], which increases the risk of falls [9].

Previous studies reported that the less-affected side assumes the function of the more-affected side during a turning task, as a compensatory strategy between the lower limbs in PD patients and freezers [9,44]. This asymmetry in performing the turning tasks is responsible for the motor manifestation characteristics of PD patients [45]. The cause of this asymmetry is not known [46]. However, it might reflect a different vulnerability of the substantia nigra pars compacta neurons to the

factors involved in the generation of the disorder. Therefore, these results suggest that freezers with higher asymmetry may experience greater difficulty than non-freezers and controls with symmetrical motor involvement [45]. The motor manifestations are predominantly observed while performing challenging turning tasks [46].

Lee et al. [47] suggested that the more-affected side of the PD patients tends to be affected predominantly throughout disease progression. Therefore, this dominant tendency may promote greater motor deficits on the more-affected side. These results suggest that turning difficulty may be related to asymmetry between the more-affected and less-affected sides of the lower limbs [47]. Moreover, shorter step lengths may be related to weakened muscle strength, which could influence the alterations in the lower limb kinematics [48]. In addition, freezers may have greater difficulty in adapting postural control during turning and switching their motor pattern to meet the asymmetrical demands while turning in the direction of the more-affected side [5]. These results suggest that the efforts made by PD patients and freezers to increase the postural stability during the turning tasks may help compensate the reduced lower limb strength of the more-affected side [48].

The findings presented in this study have some important implications. First, PD patients and freezers might experience greater difficulty in turning toward the direction of the more-affected side, which may be useful in the clinical assessment of the disease severity and FOG. Thus, fall prevention training may require focus on improving the turning performances, particularly turning toward the more-affected side, by practicing wider turns, turning more slowly, or practicing to sustain a wider base of support. Second, the ML RMS distance of the COM was shown as a marker that can distinguish between the severity of the motor symptoms in PD patients and controls, or freezers and non-freezers.

However, our study presents some limitations. First, the sample size was small for the generalization of the results obtained. Although a different sample size was used in this study, the lack of influences may have been obtained in the results. Second, the comparison of antiparkinsonian medication effects was not performed while evaluating the turning tasks of this study. Third, this study was based on difficult turning tasks that do not appear in daily life activities. Finally, this study, which used 39 markers, was considered to observe turning movements such as upper and lower limb coordination and posture control and dynamic stability of PD patients. However, we did not consider these results of upper limbs movement in our study because we focused on only lower limb movements.

### **5. Conclusions**

Our study investigated the characteristics of turning performance in the direction of the more-affected side in PD patients in comparison with controls, and in freezers in comparison with non-freezers during 360◦ and 540◦ turning tasks at the maximum speed. We found that the PD patients and freezers exhibited higher turning difficulty during challenging turning tasks such as those requiring increasing turning angles and speeds. Further, PD patients and freezers may be in greater danger of falling while turning in the direction of the more-affected side. Therefore, turning tasks involving 360◦ and 540◦ turning tasks in the direction of the IMA and OMA at the maximum speed may be useful for evaluating the turning characteristics to distinguish between PD patients and freezers. Further studies are needed to evaluate turning characteristics according to motor symptoms in PD patients at more various conditions during turning tasks.

**Ethics approval and consent to participate:** All procedures performed in studies involving human participants were performed in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. All subjects gave their informed consent for inclusion before they participated in the study. The study was approved by the Institutional Review Board of Dong-A University Hospital (IRB number: DAUHIRB-17-033; see "Ethical approval letter IRB").

**Supplementary Materials:** The following are available online at http://www.mdpi.com/1424-8220/20/11/3098/s1, Supplementary File S1: IRB number, Supplementary File S2: Research data (raw data), Supplementary Table S1: List of levodopa-equivalent doses.

**Author Contributions:** Conceptualization, H.P., C.Y., M.L., B.N., and S.-M.C.; methodology, H.P., C.Y., M.L., B.N., and S.-M.C.; formal analysis, H.P., C.Y., M.L., B.N., and S.-M.C.; investigation, H.P., C.Y., M.L., and B.N.; resources, H.P., M.L., B.N., and S.-M.C.; data acquisition, H.P., C.Y., M.L., B.N., and S.-M.C.; writing—original draft preparation, H.P., C.Y., and S.-M.C.; writing—review and editing, H.P., C.Y., M.L., B.N., and S.-M.C. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by Dong-A University research fund. This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

**Acknowledgments:** The authors would like to thank the Biomechanics Laboratory staff at Dong-A University for their assistance with data collection.

**Conflicts of Interest:** The authors declare no conflict of interest.

**Availability of data and materials:** The datasets generated and/or analyzed during the current study are not publicly available due to intellectual property reasons, but these are available upon a reasonable request.

### **References**

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

*Article*

## **Simultaneous Hand Gesture Classification and Finger Angle Estimation via a Novel Dual-Output Deep Learning Model**

### **Qinghua Gao** †**, Shuo Jiang** † **and Peter B. Shull \***

The State Key Laboratory of Mechanical System and Vibration, School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China<EMAIL_ADDRESS>(Q.G<EMAIL_ADDRESS>(S.J.)

**\*** Correspondence<EMAIL_ADDRESS>
† These authors contributed equally to this work.

Received: 21 April 2020; Accepted: 22 May 2020; Published: 24 May 2020

**Abstract:** Hand gesture classification and finger angle estimation are both critical for intuitive human–computer interaction. However, most approaches study them in isolation. We thus propose a dual-output deep learning model to enable simultaneous hand gesture classification and finger angle estimation. Data augmentation and deep learning were used to detect spatial-temporal features via a wristband with ten modified barometric sensors. Ten subjects performed experimental testing by flexing/extending each finger at the metacarpophalangeal joint while the proposed model was used to classify each hand gesture and estimate continuous finger angles simultaneously. A data glove was worn to record ground-truth finger angles. Overall hand gesture classification accuracy was 97.5% and finger angle estimation *R*2 was 0.922, both of which were significantly higher than shallow existing learning approaches used in isolation. The proposed method could be used in applications related to the human–computer interaction and in control environments with both discrete and continuous variables.

**Keywords:** modified barometric sensor; gesture recognition; convolutional neural network; dual-output model

### **1. Introduction**

With the rapid development of computer technology and the widespread use of sensors, there is an emerging demand for more intuitive human–computer interaction (HCI) [1]. HCI approaches have been investigated for various applications, including touch interaction, voice control, and emotion perception. Gesture recognition is a promising HCI area, including a wide range of potential applications: sign language recognition [2], navigation and manipulation in virtual environments [3], automotive manufacturing [4], and control for home service robots [5]. The hands are a logical choice for gesture recognition because of their dexterity, and intricate movements involving multiple degrees of freedom [6–8].

Hand gestures can be classified as static or dynamic. Static hand gesture recognition considers the shape characteristics of the hand at a point in time, and dynamic hand gestures focus on a series of hand movements over a time interval. Dynamic hand gestures can involve both hand gesture classification and finger angle estimation. Hand gesture classification can decode human intentions as discrete commands for interacting with and controlling physical hardware. Finger angle estimation can be utilized to achieve the continuous operation of physical devices. For example, varying finger angles could be mapped to operate dynamic realistic tasks, such as game joysticks in virtual reality.

Various machine learning-based algorithms have been proposed for hand gesture classification. Jiang et al. [9] recognized four surface gestures and eight air gestures based on a fusion of surface

electromyography and inertial measurement unit signals, utilizing linear discriminant analysis (LDA). Zhang et al. [10] presented Tomo, which employed electrical impedance tomography to evaluate two gesture sets with a support vector machine (SVM). Jiang et al. [11] classified singer-finger gestures and American Sign Language 0−9 number gestures with LDA, k Nearest Neighbor (KNN), and random forests (RF). Deep learning has also been studied in hand gesture classification. Wang et al. [12] proposed a combination of convolutional and recurrent neural networks (RNN) with Google's Soli sensor to recognize 11 dynamic gestures. Kim et al. [13] used Doppler radar to recognize hand gestures with a convolutional neural network (CNN).

Machine learning approaches have also been proposed for finger angle estimation. Kawaguchi et al. [14] detected finger movement from wrist deformation and used multiple regression models to estimate finger joint angles. Zhou et al. [15] estimated finger motion through forearm muscle deformation measured by A-mode ultrasound-sensing approaches with a linear ridge model and a least-squares SVM model. Deep learning has also been investigated in the continuous angle estimation of fingers. Smith et al. [16] asynchronously decoded metacarpophalangeal (MCP) joint angles of five fingers using a backpropagation neural network classifier with a Levenberg–Marquardt training algorithm. Hioki et al. [17] estimated proximal interphalangeal finger joint angles by streaming feedback into a recurrent artificial neural network with time delay factors.

Some have also proposed simultaneous hand gesture classification and finger angle estimation. Pan et al. [18] proposed a switching regime, including one LDA classifier to identify wrist movements and two state space models to continuously decode the finger angle from electromyogram (EMG) signals.

Some studies simultaneously output the results of classification and regression for other areas. Abraham et al. [20] presented a joint regression and classification framework to predict the future values of a zero-inflated time series. A linear regression model was used to predict the value of the time series, and the regression results were input to the SVM classifier to determine whether the predicted value should be adjusted to zero. Wang et al. [21] proposed a hybrid CNN model integrating a classification network and a regression network for object pose estimation from a single image, but their algorithms were based on static image data and thus not suitable for low-dimensional and dynamic time-series signals in our application.

However, current approaches suffer from several limitations. First, previous machine learning models produced limited accuracy because features were typically extracted independently from time-series signals in an ad-hoc manner [22], and the correlations among different signals were often overlooked. Although deep learning avoids the tedious process of manually extracting features, it relies on proper data structures. If the dimension of data is too small, the model may overfit and lead to low accuracy and poor performance. Second, shallow learning approaches cannot output classification values and regression values simultaneously. Although some articles investigated the output of hand gesture categories and finger angle estimation at the same time, they still relied on two superimposed algorithm models: one for classification and the other for regression. But this pattern has inherent drawbacks: (1) since the model parameters are derived from the data features, the attributes of multiple models are statistically relevant, and two separate models may create system redundancies. (2) When data changes, the classification and regression models require retraining, and the parameters of the two models cannot be adjusted simultaneously. (3) In the practical prediction applications, compared to the parallel structure that simultaneously derives the category and the regression values, the ordinal structure of first predicting the category and then estimating the angle values increases the complexity and inconvenience of the system.

Therefore, an improved compact model that can fully explore the spatial-temporal features should be built to enable two simultaneous outputs. To the best of our knowledge, there are currently no approaches for simultaneous outputs of hand gesture classification and finger angle estimation based on deep learning.

The purpose of this paper is to address the previous two limitations by introducing a novel framework to simultaneously detect hand gestures during finger movement with high accuracy, including identifying which finger is flexing and determining the joint angle of each finger. To improve the accuracy of the model, we present a new spatial-temporal method to explore the hidden relationships between different sensors and useful information contained in historical data. For simultaneous hand gesture classification and finger angle estimation, we propose a dual-output model structure. We hypothesized that the framework would enable classification of the hand gestures and estimation of finger angles simultaneously via a single deep learning model with dual outputs, and this framework would have better performance than single-output, shallow learning models.

### **2. Materials and Methods**

### *2.1. Wristband Prototype and Experiment Protocol*

### 2.1.1. Wristband prototype

While current wearable approaches are limited because of low sensitivity or poor linearity, our prototype takes advantage of the fact that hand motion changes the shape of the wrist due to the expansions and contractions of the muscles, which brings position and thickness transformations of tendons and muscles, producing deformations on the surface contour of the wrist. We thus used a wristband with ten embedded modified barometric sensors with high sensitivity and excellent linearity to measure the activity of the tendons and muscles that cause finger movement [23]. The deformations apply pressure to the surface of modified barometric sensors. When a subject with the wristband performs various gestures, a corresponding contact pressure distribution can be obtained from the wristband (Figure 1).

**Figure 1.** System framework of the novel dual-output convolutional neural network (CNN) model for simultaneous hand gesture recognition and finger angle estimation.

### 2.1.2. Gesture Set

When handling actual tasks by hand, multiple fingers often work in concert. The Taiwanese Sign Language includes 50 fundamental gestures [24], and many of them are the linkage of multiple fingers. Most studies about interlocking multiple fingers used static positions. However, we studied the dynamic process of finger movement. The dynamic recognition process of multiple fingers is complicated because the flexion angles of different fingers are different, and it involves a lot of research on hand anatomy. Multi-finger movements include thumb opposition, multi-finger flexion/extension, multi-finger adduction/abduction, etc. These multi-finger movements can be decomposed into multiple single-finger movements. Analyzing single-finger movements is the basis of multi-finger movements. At present, most studies on finger movement are based on single finger movement [15,25]. Therefore, we chose 5 single fingers as the gesture set to identify the moving fingers and estimate the finger joint angle.

### 2.1.3. Experiment Protocol

Ten subjects (8 male, 2 female, age: 22–26 year, height: 166–184 cm, weight: 57–76 kg, wrist circumference: 15.3–16.9 cm) participated in the experiment. All subjects gave their informed consent in accordance with the Declaration of Helsinki before they participated in the study. During each trial, subjects flexed/extended one finger at the MCP joint repeatedly at their self-selected frequency. The minimum and maximum angles of thumb were required to be 0◦ and 55◦, and of other 4 fingers were required to be 0◦ and 90◦. Some subjects could not bend their fingers to 90◦ [26]. Therefore, the actual maximum and minimum angles were decided by the subjects themselves. Fingers were flexed in the order of thumb, index finger, middle finger, ring finger, and pinky finger, and the data were encoded into 5 categories and recorded. We used a data glove (5DT Data Glove 14 Ultra, South Africa, Gauteng, Pretoria), which is widely used and regarded as the gold standard for measurement in the estimation of finger joint angles, to record the real finger angle relative to the ground at the same time [14]. Each finger was flexed/extended for 25 s, there were 5 fingers per trial (Figure 2), and the sampling frequency was 40 Hz, so 25 × 5 × 40 = 1000 samples were collected per trial. Each subject performed 10 trials [27], so 10 × 10 = 100 trials were performed. Thus, there was a total of 1000 × 100 = 100, 000 samples.

**Figure 2.** Raw signals from 10 modified barometric sensors for a typical trial (subject 1, trial 1) showing distinct patterns between each gesture.

For each trial, we had 10 columns of modified barometric sensing data, corresponding finger categories and finger angle data. The data dimension was (1000, 12). Given these data, we aimed to learn a mapping *f* from the pressure sensing data *x* ∈ *X* to dual outputs [*y*1, *y*2] ∈ *Y*, where *y*1 represents the label of the finger, totaling 5 classes, and *y*2 represents the corresponding MCP angle measured by the data glove.

#### *2.2. Data Augmentation*

Sequential data were usually represented as a multivariate time series, and so the 10 columns of modified barometric sensing data were encoded as 10 features. However, there are restrictions on this data form. According to our sensing scheme and physiological structure of users, spatial locations between different sensors are related, but multivariate time-series form cannot distinguish a relationship between sensors that are far away. Suppose there are N-channel signals arranged in columns. For the column *k* signal, it is only adjacent to the column *k* − 1 signal and column *k* + 1 signal. When referring to the spatial information of the column *k* signal, only the correlation between column *k* signal and the column *k* − 1 signal, and the correlation between the column *k* signal and column *k* + 1 signal are considered, while the spatial positional relationships with signals of other columns are ignored. Therefore, we propose a data augmentation method to explore the hidden spatial–temporal relationships between different sensors.

We propose to convert time-series signals collected from sensors to new movement images that contain hidden relationships between any pair of signals (Figure 1). Features would be extracted from the newly generated movement image, providing additional potential spatial information between the signals.

The algorithm steps are as follows:

Our signal data have 10 signal sequences [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]. After Step 1 and Step 2, a movement image is generated with the size of 42 × *Ls*, where *Ls* is the time length of signal sequences. For each trial, *Ls* = 1000. The 42-dimensional vectors are [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 1, 3, 5, 7, 9, 1, 4, 6, 8, 10, 2, 4, 7, 10, 3, 6, 9, 2, 5, 8, 1, 5, 9, 3, 7, 1, 6, 10, 4, 8, 2, 6], and each column from 1 to 10 is adjacent to the other nine columns. A time window is utilized to crop the signals based on Step 3. The window length is 42, and the size of our movement image is finalized as 42 × 42. The window slides 1-time step along the time axis to generate a new movement image, and after Step 4, *Ls* − 42 images are ultimately obtained. These images are new samples of the model, and the dimension of the samples is (*Ls* − 42, 42, 42).

For each newly generated movement image, there are 42 finger labels and angles corresponding to window length. We select the finger category with the highest frequency as the label for this movement image, and the finger angle of the last time step as angle regression value for this image. For speeding up the convergence, the sensor features and finger angle data are standardized with a mean of 0 and a standard deviation of 1.

### *2.3. Model Architecture*

We proposed a deep learning model with dual outputs to enable the classification of finger gestures and the estimation of finger angles simultaneously (Figure 3). We built a dual-output model based on CNN. We used 10 sensor units to measure the pressure changes of 10 tendons related to the finger movement. The pressure can be converted into a force myography (FMG). CNN can be used to identify specific FMG patterns. CNN is a deep learning model that extracts local features of data through convolution operations and weight sharing and is widespread in computer vision and pattern recognition.

**Figure 3.** Dual-output CNN model structure diagram.

### 2.3.1. Convolutional Layer

The convolutional layer includes multiple convolutional filters that extract the signal features by the convolution process. The sparse connection and weight sharing of the convolution kernels allow the convolutional layer to learn features with a small amount of calculations.

An activation function is used after the convolutional filter to solve non-linear problems. Deep learning currently employs a Rectified Linear Unit (ReLU) function as activation function, *f*(*x*) = *max*(0, *x*). This function defines the non-linear output after the linear transformation of the neuron.

The pooling layer is used for reducing data dimensions and preventing network overfitting after the convolutional layer. Several convolutional layers and pooling layers are often stacked to extract gradually more abstract features.

### 2.3.2. Dual Outputs

A flatten layer is utilized to transform a multi-dimensional array to a one-dimensional vector, which then flows into a dense layer to reduce the dimension of neurons. Finally, the model takes two parallel dense layers as outputs: one with *K* neurons to export the category attribute, and the other with 1 neuron to export the regression attribute, where *K* is the number of hand gesture categories (Figure 3).

The dense layer used for classification requires a *K*-way softmax activation function to obtain the probability distribution over different categories:

$$y\_i = \frac{e^{z\_i}}{\sum\_{j=1}^{K} e^{z\_j}}, \; i = 1, 2, 3, \dots, K \tag{1}$$

where *z* is the output vector of the dense layer, *yi* is the probability that the sample belongs to the category *i*. The category with the highest probability is the predicted label of the sample.

### 2.3.3. Loss Weighting and Training Algorithm

Angle estimation is a scalar regression task, while hand gesture recognition is a multi-category task. The training model needs to assign different loss functions to the two outputs. The mean-squared error can be used as the loss of regression output:

$$loss\_r = MSE = \frac{1}{n} \sum\_{i=0}^{n-1} (y\_i - \mathcal{y}\_i)^2 \tag{2a}$$

where *lossr* is the regression loss, *yi* is the true value of sample *i*, *y*ˆ*i* is the predicted value of sample *i*, and *n* is the number of samples.

Categorical cross entropy can be used as the loss of classification output:

$$\text{loss}\_{\mathbf{c}} = J = -\frac{1}{n} \sum\_{i=0}^{n-1} \sum\_{j=0}^{K-1} y\_{i,j} \log(p\_{i,j}) \tag{2b}$$

where *lossc* is the classification loss, *yi*,*j* ∈ (0, 1) indicates whether the sample *i* is predicted to be the label *j*, *pi*,*j* is the probability that the sample *i* is predicted to be the label *j*, and *K* is the number of hand gesture categories.

Two losses need to be combined into a total loss. The scale of the two losses is different, which leads to the total loss being dominated by one of the tasks, and the loss of other task cannot affect the learning process of the network. A practical way to solve the problem is loss weighting, and thus, we weighted the losses to get a global loss. The goal of model optimization is to minimize total loss during training.

$$
\log\_{total} = w\_{\text{closs}\_{\text{c}}} + w\_{\text{rloss}\_{\text{r}}} \tag{2c}
$$

where *losstotal* is a total loss, *wc* is the weight of classification loss, and *wr* is the weight of regression loss. The ratio of *wr* to *wc* is the weight ratio, which is a hyperparameter.

The coefficients of convolution filters and dense layers are obtained by training a dataset. The dual-output model is multi-task learning based on parameter sharing. Multiple tasks share the representation at shallow layers, which can reduce network overfitting and improve the generalization effect.

Our dual-output CNN model was established according to the principle described above and was trained in Keras 2.2. The first convolutional layer filtered the 42 × 42 input movement image with 32 kernels of size 3 × 3, followed by a max-pooling layer of size 2 × 2. The second convolutional layer filtered the output of the first pooling layer with 32 kernels of size 3 × 3, followed by a 2 × 2 max-pooling layer. The flatten layer vectorized the output into a 128-dimension feature vector. The dense layer consisted of 32 units. The network ended with a 5-way dense layer as classification output and a 1-way dense layer as the regression output. We adopted: (1) a ReLU activation function in each hidden layer, (2) a Softmax function in the 5-way dense layer, (3) a 4:1 weight ratio between the finger angle regression loss and the finger gesture classification loss, and (4) an RMSprop function as the optimizer with a batch size of 32 and 10 epochs in model training.

### *2.4. Model Performance Evaluation*

Leave-one-trial-out cross validation was utilized to assess the performance of the model. For each subject, the network was trained based on his training subset and evaluated based on his test subset. Each subject had 10 sets of experimental data, and we conducted 10 verifications for each subject. Each verification used 9 sets of data as the training set, the remaining set of data as the test set. Therefore, for the training set, the time length *Ls* = 9 × 1000 = 9000, and the dimension of features was (9000 − 42, 42, 42). For the test set, the time length *Ls* = 1000, and the dimension of the features was (1000 − 42, 42, 42). The test set selected for each verification was different from each other.

For each subject, the proportion of correctly recognized finger gestures in all test samples was used as the accuracy index of finger classification:

$$Accuracy = \frac{n\_{\text{correct}}}{n\_{\text{total}}} \tag{3}$$

where *ncorrect* is the number of samples classified correctly, and *ntotal* is the number of all test samples. The *R*2 was used to access the accuracy of finger angle estimation:

$$R^2 = 1 - \frac{\sum\_{i} (y\_i - \hat{y}\_i)^2}{\sum\_{i} (y\_i - \overline{y\_i})^2} \tag{4}$$

where *yi* is the true value, *y*ˆ*i* is the predicted value, and *yi* is the mean value of test samples. The larger the *R*2 score, the better the model fitting effect. The average accuracy over all subjects was calculated as the ultimate gesture classification accuracy and the average *R*2 score over all subjects was calculated as the ultimate *R*2 score.

### *2.5. Benchmarking against Other Machine Learning Models*

For validating the performance of our proposed dual-output model, we established the following benchmarks: CNN without data augmentation, RNN, single-output classification models, and single-output regression models. SVM and RF can be utilized for both classification and regression, so they were chosen as the benchmarks of the single-output models.

A paired *t*-test was used to assess whether there were significant differences between any benchmark and the proposed method. Since there were only 10 subjects, and the size of the samples was small, the data might not conform to the normal distribution. The nonparametric test does not require a population distribution, so a nonparametric paired *t*-test was ultimately selected. SPSS was used to perform the nonparametric paired *t*-test on the results of 10 subjects. When *p* < 0.05, we considered it a significant difference between the two groups of samples.

### 2.5.1. CNN without Data Augmentation

A dual-output CNN without data augmentation model was established to observe the impact of our proposed data augmentation method on the dual-output deep learning model. The original signal was composed of 10 columns of features, and we took a time window with a length of 10. The size of the movement image was finalized as 10 × 10, and the dimension of the samples was (*Ls* − 10, 10, 10). Other selections, namely (1) the processing of finger labels and angles to keep up with each image, (2) the structures and parameters of the dual-output model, (3) the parameters of model training, were the same as the CNN with data augmentation model.

### 2.5.2. RNN

RNN is a deep learning model with memory and is widely used to process time-series signals. Base on the gating mechanism, long short-term memory (LSTM) retains long-term sequence information and alleviates the gradient disappearance problem [29]. The gated recurrent unit (GRU) is a simplified variant of the LSTM [30]. The GRU network does not maintain additional state vectors and has less computation and faster training than the LSTM.

We built a dual-output model based on GRU to compare with the CNN dual-output model. The network structure and parameters were as follows: (1) the 10 columns of original signals were taken as the input, so the dimension of the input was 10, (2) the length of time step was 10, (3) the finger category with the highest frequency in 10-length time step was taken as the true label, and the last angle value in 10-length time step was taken as true angle value, (4) 1 GRU layer was used, (5) the dimension of the output was 32, (6) the dropout of input unit was 0.2, and the dropout of the recurrent unit was 0.2.

The GRU layer was followed by the dual-output structure, which was the same as the principle described in the dual-output CNN model. Other selections of the weight ratio, the optimizer, the batch size, and the epoch, were the same as the CNN model parameters.

#### 2.5.3. SVM

SVM is a widely adopted approach in the realm of hand gesture recognition. It uses hinge loss to calculate empirical risk and uses regularization to optimize structural risk. SVM can be used for classification with support vector classification (SVC) and regression with support vector regression (SVR). SVC is a classifier with sparsity and robustness. It can be nonlinearly classified by the kernel method. Its goal is to maximize the margin between support vectors and the hyperplane. Different from SVC, the goal of SVR is to minimize the total deviation of all sample points from the hyperplane.

We used the RBF kernel function and tuned the penalty coefficient (*C*) and the kernel parameter gamma (γ) to obtain higher accuracy and *R*2 score. An SVC model with the parameters *C* = 5 and γ = 0.1 was established to classify finger gestures, and an SVR model with parameters *C* = 10 and γ = 0.1 was established to estimate the finger angle.

### 2.5.4. RF

RF is a bagging-based ensemble learning method composed of multiple weak learners. RF can be used for classification with RF classifier and regression with RF regressor. RF classifier randomly generates k decision trees from the training set to form a random forest by the bootstrap resampling technique. The classification result is determined by decision tree voting scores. Different from classification, the RF regression result is the average of the outputs of decision trees.

We tuned the parameters of the number of decision trees and the maximum number of features selected when dividing leaf nodes. For the RF classifier, the number of decision trees was 100, and the maximum number of features was √*nf eatures*, where *nf eatures* is the number of total features. For the RF regressor, the number of decision trees was 60, and the maximum number of features was *log*2*nf eatures*.

### *2.6. The Impact of Sensor Numbers on the Dual-Output Model*

We compared the results of different sensor numbers between 1 and 10. Tree-based feature selection was used to calculate the importance of features. The RF classifier and RF regressor were, respectively, applied as tree models to obtain the feature importance α*c* on classification attributes and α*r* on regression attributes. The mean value α*i* of α*c* and α*r* for each subject was taken as the total importance of this subject, where *i* = 1, 2, ... , 10. We calculated the average of total importance for all subjects as the ultimate results of feature importance, which was sorted in reverse order as [8, 1, 3, 7, 5, 6, 2, 4, 9, 10]. We selected 1 to 10 features with the highest feature importance and used the proposed data augmentation method and dual-output model to obtain the results of different sensor numbers.

### **3. Results**

The mean classification accuracy of hand gesture was 97.5% across the subjects. A confusion matrix was drawn from subject 10 (Figure 4), and there was no obvious confusion in the classification. The thumb had the highest classification accuracy at 100.0%, while the middle finger had the lowest classification accuracy at 97.5%. The predicted label typically confused the middle finger and ring finger. The mean *R*2 score of the finger angle estimation over all subjects was 0.922. The fitting effect was better in the rising and falling phases of the curve, but there were deviations in the peaks and valleys (Figure 5).

**Figure 4.** Confusion matrix of hand gesture classification from a typical subject (subject 10). The overall classification accuracy for this trial was 99.2%.

**Figure 5.** Typical trial showing actual and predicted normalized finger angles (subject 10, thumb). The *R*2 score for this trial was 0.965. Finger angles were normalized from 0 and 1.

The results (Figure 6) of four benchmarks were as follows: (1) CNN without data augmentation had a classification accuracy of 96.3% and angle *R*2 score of 0.916. (2) RNN had a classification accuracy of 95.6% and an angle *R*2 score of 0.881. (3) SVM had a classification accuracy of 96.5% and an angle *R*2 score of 0.887. (4) RF had a classification accuracy of 95.8% and an angle *R*2 score of 0.905.

**Figure 6.** Overall results for different models: CNN\_Aug (presented model) = convolutional neural network with data augmentation, CNN = convolutional neural network without data augmentation, RNN = recurrent neural network, SVM = support vector machine, and RF = random forest. (**a**) The results of hand gesture classification accuracy for different models. (**b**) The results of finger angle estimation fitting for different models. ∗∗ denotes *p* < 0.001 and ∗ denotes *p* < 0.05.

The results (Figure 7) of different sensor numbers between 1 and 9 were as follows: (1) one sensor had a classification accuracy of 76.4% and an angle *R*2 score of 0.528; (2) two sensors had a classification accuracy of 87.8% and an angle *R*2 score of 0.729; (3) three sensors had a classification accuracy of 91.5% and an angle *R*2 score of 0.820; (4) four sensors had a classification accuracy of 93.2% and an angle *R*2 score of 0.855; (5) five sensors had a classification accuracy of 95.7% and an angle *R*2 score of 0.889; (6) six sensors had a classification accuracy of 96.9% and an angle *R*2 score of 0.909; (7) seven sensors had a classification accuracy of 97.3% and an angle *R*2 score of 0.916; (8) eight sensors had a classification accuracy of 97.3% and an angle *R*2 score of 0.918; (9) nine sensors had a classification accuracy of 97.4% and an angle *R*2 score of 0.920.

**Figure 7.** Overall results for different sensor numbers between 1 and 10. (**a**) The results of hand gesture classification accuracy for different sensor numbers. (**b**) The results of finger angle estimation fitting for different sensor numbers.

### **4. Discussion**

### *4.1. Comparison with Previous Results*

This paper presents a novel framework to output hand gesture classification and finger angle estimation simultaneously via a deep learning model with dual outputs. Compared with previous methods, this scheme enables outputs of both classification and regression attributes simultaneously with higher accuracy. For hand gesture recognition with shallow learning, Al-Timemy et al. [31] achieved 98% accuracy for the classification of 15 classes of different finger movements with time-domain autoregression, orthogonal fuzzy neighborhood discriminant analysis, and LDA. For hand gesture recognition with deep learning, Geng et al. [32] used EMG images and a deep convolutional network for an 8-gesture within-subject recognition. The recognition accuracy reached 89.3% on a frame of the image and 99.0% using the majority voting algorithm over 40 frames. For finger angle estimation with shallow learning, Huang et al. [25] employed an ultrasound image to predict the MCP joint angles of all fingers in nonperiod movements and got an average correlation of 0.89 ± 0.07. For finger angle estimation with deep learning, Ngeo et al. [33] utilized a fast feedforward artificial neural network and a nonparametric Gaussian process (GP) regressor to capture finger angles during finger flexion and extension. This method yielded an *R*2 of 0.78. For hand gesture classification and finger angle estimation simultaneously with two superimposed models, Yang et al. [34] reported a class-specific proportional control system and utilized LDA for gesture recognition and GP regression for muscle contraction force estimation via wearable ultrasound sensing. The accuracy of this model was 93.7%, and the *R*2 was 0.927. Previous research [27] used the SVM, LDA, and KNN models for the recognition of three sets of static hand gesture, and an RF model for dynamic single finger angle estimation. Thus, the classification and regression attributes of gesture data were not considered jointly. In addition, previous research achieved a classification accuracy of 94.4% on a set of static single finger flexion gestures. However, we obtained a classification accuracy of 97.5% on a set of dynamic single finger flexion gestures. Both studies examined finger angle estimation based on a dynamic single finger flexion experiment, but we got a higher *R*2 of 0.922 compared with 0.79 in previous research. Therefore, our model implemented more functions and exhibited higher accuracy.

### *4.2. Analysis of Confusion Matrix and Fitting Curve*

The least frequent misclassification occurred in the thumb, while the most frequent misclassification occurred in the middle finger (Figure 4). This happens because the flexion and extension movements of the thumb are on the sagittal axis, while the flexion and extension movement of the other four fingers are on the coronal axis. Therefore, the flexion and extension of the thumb are rarely affected, and the classification is not easily confused with the other four fingers. When the middle finger was flexing and stretching, it was easily confused with the index finger and ring finger because the same ligament drives the index finger and ring finger at the same time.

Angle estimation did not fit as well at the peaks and troughs in some cases (Figure 5). The reason might be that the activities of the tendons and muscles that caused finger movement were very complicated when the fingers varied from a flexed state to a stretched state or vice versa, and the model could not statistically excavate the principles.

### *4.3. E*ff*ects of Hyperparameters*

The effects of hyperparameters on the performance of the model were investigated.

### 4.3.1. Window Length

The window length indicated the length of each movement image. The longer the window length, the more past information each image contained, which could improve the performance of the model. However, when the window length was too long, each image contained too much past information, which caused the model to be overburdened and make it overfit. Therefore, the appropriate window

length should be selected. In the model training, we found that the model performance was best when the window length was between 40 and 50. Considering that the channel dimensions after data augmentation were 42, the window length was ultimately selected to be 42.

### 4.3.2. Network Structure of the Model

We also assessed the impact of the network structure on the performance of the model and found that, with the increases in the model depth, the performance did not improve significantly. Therefore, a shallow network structure was suitable because a very large and deeper network structure would learn noise and other interference errors and could result in performance degradation or increased computational cost and network burden.

### 4.3.3. Batch Size and Epoch

The batch size and the epoch followed a similar trend that, as they increased, the performance of the model decreased. So, a smaller batch size and epoch should be selected. The reason for the shallow network structure and small batch size and epoch might be that our data was less complicated, so a lightweight network was sufficient to extract features.

### 4.3.4. Weight Ratio

The weight ratio was a hyperparameter introduced to make the scales of classification loss and regression loss close. In the model training, we adjusted the weight ratio from 20:1 to 1:20 and found fluctuations in performance. A possible reason for this is related to the decay rate of the loss function. In the model training, the decay rates of the two loss functions were inconsistent. Therefore, a fixed weight ratio might not be effective enough. In the weight ratios we tested, when the weight ratio was 4:1, the effect was best, so we used 4:1 as the final weight ratio.

### *4.4. Feature Extraction in Sequential Signals and Image Signals for CNN*

### 4.4.1. Feature Extraction in Sequential Signals for CNN

CNN can be well applied to sequential signals, such as inertial measurement units (IMU) signals and audio signals. Time can be viewed as a spatial dimension. The 1D convolution layer can extract and recognize local patterns in a sequence. Because the same transformation is performed on every patch, a pattern learned at a specific position can be recognized at a different position later, making 1D CNN translation-invariant for temporal translations.

### 4.4.2. Feature Extraction in Converted Image Signals for CNN

When the time series data are converted into images, the convolutional kernel of 2D CNN slides on the image. The temporal information is extracted when the convolutional kernel slides from top to bottom on the image. The spatial information between different sensors is extracted when the convolutional kernel slides from left to right on the image.

When the convolutional kernel slides from top to bottom on the image, it is very similar to 1D CNN. When the convolutional kernel slides from left to right on the image, and the data of one sensor changes to the data of an adjacent sensor, a drastic change will occur, thus generating a high-frequency signal. Through the convolution operation between the convolution kernel and the high-frequency signal, the frequency information will be selected, so CNN can extract the position information between the sensors.

### *4.5. Comparison with Other Machine Learning Models*

From the comparison of the dual-output models based on CNN with data augmentation and other models (Figure 6), we found that the model built by CNN with data augmentation performed best. This model had the highest accuracy, highest *R*2 score, and the smallest deviations.

### 4.5.1. Comparison between Dual-Output Models with and without Data Augmentation

From the comparison between the dual-output CNN models with and without data augmentation (Figure 6), for hand gesture classification, the CNN without data augmentation model had a classification accuracy of 96.3%, and the CNN with data augmentation model had a classification accuracy of 97.5%. A nonparametric paired *t*-test indicated significant differences between the CNN with data augmentation and the CNN without data augmentation. The data augmentation method not only made classification accuracy higher but it also made deviation smaller. For finger angle estimation, the CNN without data augmentation model had a *R*2 score of 0.916, and CNN with data augmentation model had a *R*2 score of 0.922. Although a nonparametric paired *t*-test indicated no significant difference between dual-output models with and without data augmentation, there was a smaller deviation in the CNN with data augmentation model.

The results show that the data augmentation method achieved better results. This is because the data augmentation method took into account the hidden relationships between different sensors and useful information contained in history data.

### 4.5.2. Comparison between Dual-Output Models Based on CNN with Data Augmentation and RNN

From the comparison between dual-output models based on CNN with data augmentation and RNN (Figure 6), the RNN model had a classification accuracy of 95.6% for hand gesture classification and had an *R*2 score of 0.881 for finger angle estimation. A nonparametric paired *t*-test indicated significant differences between the CNN with data augmentation model and RNN model both in hand gesture classification and finger angle estimation. CNN with the data augmentation model had a smaller deviation. Similarly, CNN without the data augmentation model was also better than the RNN model. Therefore, the models built by CNN performed better than RNN.

RNN is mainly utilized to describe the continuous output in time and to connect spatial states together. It is more suitable for predicting future data from past data, such as the prediction of the next moment of the video, the prediction of the context of the document, and the prediction of future temperature. CNN, however, is more suitable for describing the state of a certain space and feature extraction. Our research tends to perform feature extraction on data to identify patterns in gesture data, rather than predicting what gesture will be made or what angle the finger will bend at the next moment.

RNN only considers the temporal information of the data and does not consider the spatial information. We have 10 sensors around the wrist, and there are positional relationships between the 10 sensors. CNN can extract the position information between the sensors.

CNN is more appropriate for hand gesture classification and finger angle estimation by the above discussion. The results further show that although a dual-output model could be implemented based on deep learning, our proposed method had better results.

### 4.5.3. Comparison between Dual-Output Models and Single-Output Models

From the comparison between the dual-output models based on deep learning and the single-output models based on other machine learning (Figure 6), we found that the performance of the dual-output models based on deep learning was not necessarily higher than the single-output model based on shallow learning, and sometimes even worse. For example, in terms of finger angle estimation, the result of CNN without data augmentation was higher than SVM and RF, but the result of RNN was much lower than RF. In terms of hand gesture classification, although the result of CNN was higher than RF, the result of RNN was much lower than SVM. However, in general, our proposed method was superior to shallow learning models in both hand gesture recognition and finger angle estimation.

The results of the above comparisons can verify our hypothesis. The newly proposed data augmentation method can discover hidden spatial position relationships in time-series signals and improve the accuracy of the results. Deep learning models can learn different attributes of the signal at the same time rather than in succession. When the model adjusts its internal features, other features associated with them can automatically adapt and adjust. This is more powerful than stacked shallow learning models.

### *4.6. Comparison of Results with Di*ff*erent Sensor Numbers*

The classification accuracy and angle fitting degree improved with the increase in sensors (Figure 7). When using the optimal six or more sensors, the classification accuracy reached more than 96% and the *R*2 value reached more than 0.9. When using the optimal three to five sensors, the classification accuracy reached more than 90% and the *R*2 value reached more than 0.8. When using the optimal one or two sensors, the results dropped sharply, and the model performance was poor. Therefore, the performance of the model could be improved by increasing the number of sensors. However, when the optimal number of sensors exceeded six, the performance of the model improved more slowly, and the device became more complicated.

#### *4.7. Calculation Time and Real-Time Application*

In practical application, the time duration for each gesture was about 2 to 3s. Therefore, after the dual-output model was trained offline, we selected samples of 2 s in a test set (subject 10, trial 1) to calculate the prediction time. We got the prediction time 0.0444 s on the proposed dual-output model (Table 1). For gesture recognition, when the time delay was less than 0.08 s, the model could be employed in a real-time system [35]. Therefore, we think the dual-output model could be used in real-time. For the other two dual-output model benchmarks, their time delays were also less than 0.08 s (Table 1), so they could also be used in real-time systems.

### *4.8. Improvement of Dual-Output Model*

A loss weighting method was used to define the total loss in our dual-output model, but this method introduced a hyperparameter, and the fixed weight ratio might not perform well due to the different decay rates of loss functions. Therefore, an adaptive weight variable could be introduced to replace the weight ratio.