*Article* **3D Skeletal Joints-Based Hand Gesture Spotting and Classification**

**Ngoc-Hoang Nguyen, Tran-Dac-Thinh Phan, Soo-Hyung Kim, Hyung-Jeong Yang and Guee-Sang Lee \***

Department of Artificial Intelligence Convergence, Chonnam National University, 77 Yongbong-ro, Gwangju 500-757, Korea<EMAIL_ADDRESS>(N.-H.N<EMAIL_ADDRESS>(T.-D.-T.P<EMAIL_ADDRESS>(S.-H.K<EMAIL_ADDRESS>(H.-J.Y.)

**\*** Correspondence<EMAIL_ADDRESS>
**Abstract:** This paper presents a novel approach to continuous dynamic hand gesture recognition. Our approach contains two main modules: gesture spotting and gesture classification. Firstly, the gesture spotting module pre-segments the video sequence with continuous gestures into isolated gestures. Secondly, the gesture classification module identifies the segmented gestures. In the gesture spotting module, the motion of the hand palm and fingers are fed into the Bidirectional Long Short-Term Memory (Bi-LSTM) network for gesture spotting. In the gesture classification module, three residual 3D Convolution Neural Networks based on ResNet architectures (3D\_ResNet) and one Long Short-Term Memory (LSTM) network are combined to efficiently utilize the multiple data channels such as RGB, Optical Flow, Depth, and 3D positions of key joints. The promising performance of our approach is obtained through experiments conducted on three public datasets—Chalearn LAP ConGD dataset, 20BN-Jester, and NVIDIA Dynamic Hand gesture Dataset. Our approach outperforms the state-of-the-art methods on the Chalearn LAP ConGD dataset.

**Keywords:** continuous hand gesture recognition; gesture spotting; gesture classification; multi-modal features; 3D skeletal; CNN

#### **1. Introduction**

Nowadays, the role of dynamic hand gesture recognition has become crucial in visionbased applications for human-computer interaction, telecommunications, and robotics, due to its convenience and genuineness. There are many successful approaches to isolated hand gesture recognition with the recent development of neural networks, but in real-world systems, the continuous dynamic hand gesture recognition remains a challenge due to the diversity and complexity of the sequence of gestures.

Initially, most continuous hand gesture recognition approaches were based on traditional methods such as Conditional Random Fields (CRF) [1], Hidden Markov Model (HMM), Dynamic Time Warping (DTW), and Bézier curve [2]. Recently, deep learning methods based on convolution neural networks (CNN) and recurrent neural networks (RNN) [3–7] have gained popularity.

The majority of continuous dynamic hand-gesture recognition methods [3–6] include two separate procedures: gesture spotting and gesture classification. They utilized the spatial and temporal features to improve the performance mainly in gesture classification.

However, there are limitations in the performance of gesture spotting due to its inherent variability in the duration of the gesture. In existing methods, gestures are usually spotted by detecting transitional frames between two gestures. Recently, an approach [7] simultaneously performed the task of gesture spotting and gestures classification, but it turned out to be suitable only for feebly segmented videos.

Most of the recent researches [8–11] intently focus on improving the performance of the gesture classification phase, while the gesture spotting phase is often neglected on the

**Citation:** Nguyen, N.-H.; Phan, T.-D.-T.; Kim, S.-H.; Yang, H.-J.; Lee, G.-S. 3D Skeletal Joints-Based Hand Gesture Spotting and Classification. *Appl. Sci.* **2021**, *11*, 4689. https:// doi.org/10.3390/app11104689

Academic Editor: Hyo-Jong Lee

Received: 14 April 2021 Accepted: 18 May 2021 Published: 20 May 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

assumption that the isolated pre-segmented gesture sequences are available for input to the gesture classification.

However, in real-world systems, spotting of the gesture segmentation plays a crucial role in the whole process of gesture recognition, hence, it greatly affects the final recognition performance. In paper [3], they segmented the videos into sets of images and used them to predict the fusion score, which means they simultaneously did the gesture spotting and gesture classification. The authors in [5] utilized the Connectionist temporal classification to detect the nucleus of the gesture and the no-gesture class to assist the gesture classification without requiring explicit pre-segmentation. In [4,6], the continuous gestures are often spotted into isolation based on the assumption that hands will always be put down at the end of each gesture which turned out to be inconvenient. It does not work well for all situations, such as in "zoom in", "zoom out" gestures, i.e., when only the fingers move while the hand stands still.

In this paper, we propose a spotting-classification algorithm for continuous dynamic hand gestures which we separate the two tasks like [4,6] but we avoid the existing problems of those methods. In the spotting module, as shown in Figure 1, the continuous gestures from the unsegmented and unbounded input stream are firstly segmented into individually isolated gestures based on 3D key joints extracted from each frame by 3D human pose and hand pose extraction algorithm. The time series of 3D key poses are fed into the Bidirectional Long Short-Term Memory (Bi-LSTM) network with connectionist temporal classification (CTC) [12] for gesture spotting.

**Figure 1.** Gesture Spotting-Classification Module.

The isolated gestures segmented using the gesture spotting module are classified in the gesture classification module with a multi-modal M-3D network. As indicated in Figure 1, in the gesture classification module, the M-3D network is built by combining multi-modal data inputs which comprise RGB, Optical Flow, Depth, and 3D pose information data channels. Three residual 3D Convolution Neural Network based on ResNet architectures (3D\_ResNet) [13] stream networks of RGB, Optical Flow and Depth channel along with an LSTM network of 3D pose channel are effectively combined using a fusion layer for gesture classification.

The preliminary version of this paper has appeared in [14]. In this paper, depth information has been considered together with 3D skeleton joints information with extensive experiments, resulting in upgraded performance.

The remainder of this paper is organized as follows. In Section 2, we review the related works. The proposed continuous dynamic hand gesture recognition algorithm is intently discussed in Section 3. In Section 4, the experiments with proposed algorithms conducted on three published datasets—Chalearn LAP ConGD dataset, 20BN-Jester, and NVIDIA Dynamic Hand Gesture Dataset are presented with discussions. Finally, we conclude the paper in Section 5.

#### **2. Related Works**

In general, the continuous dynamic gesture recognition task is more complicated than the isolated gesture recognition task, where the sequence of gestures from an unsegmented and unbounded input stream are separated into complete individual gestures, called gesture spotting or gesture segmentation before classification. The majority of recent researchers solve the continuous dynamic gesture recognition task using two separate processes—gesture spotting and gesture recognition [1,4–6].

In the early years, the approaches for gesture spotting were commonly based on traditional machine learning techniques for the time series problems such as Conditional Random Fields (CRF) [1], Hidden Markov Model (HMM) [2], and Dynamic Time Warping (DTW) [3]. Yang et al. [1] presented a CRF threshold model that recognized gestures based on system vocabulary for labeling sequence data. Similar to the method introduced by Yang, Lee et al. [2] proposed the HMM-based method, which recognized gestures by the likelihood threshold estimation of the input pattern. Celebi et al. [3] proposed a template matching algorithm, i.e., the weighted DTW method, which used the time sequence of the weighted joint positions obtained from a Kinect sensor to compute the similarity of the two sequences. Krishnan et al. [15] presented a method using the Adaptive Boosting algorithm based on the threshold model for gesture spotting using continuous accelerometer data and the HMM model for gesture classification. The limitations of these methods are the parameter of the model has been decided through experience and the algorithm is sensitive to noise. In the recent past, with the success of deep learning applications in computer vision, deep learning approaches have been utilized for hand gesture recognition to achieve impressive performance compared to traditional methods.

The majority of the methods using recurrent neural networks (RNN) [16–18] or CNN [8,10,19–21] focus only on isolated gesture recognition, which ignores the gesture spotting phase. After the dataset for continuous gesture spotting-Chalearn LAP ConGD dataset was provided, a number of methods have been proposed to solve both phases of gesture spotting and gestures recognition [3,4,6]. Naguri et al. [6] applied 3D motion data input from infrared sensors into an algorithm based on CNN and LSTM to distinguish gestures. In this method, they segmented gestures by detecting transition frames between two isolated gestures. Similarly, Wang et al. [3] utilized transition frame detection using two streams CNN to spot gestures. In another approach proposed by Chai et al. [4], continuous gestures were spotted based on the hand position detected by Faster R-CNN and isolated gesture was classified by two parallel recurrent neural network SRNN with RGB\_D data input. The multi-modal network, which combines a Gaussian-Bernoulli Deep Belief Network (DBN) with skeleton data input and a 3DCNN model with RGB\_D data, was effectively utilized for gesture classification by Di et al. [7]. Tran et al. [22] presented CNN based method using a Kinect Camera for spotting and classification of hand gestures. However, the gesture spotting was done manually from a pre-specified hand shape or finger-tip pattern. And classification of hand gestures used only fundamental 3DCNN networks without employing the LSTM network. The system is based on the Kinect system and the comparison using a commonly used public dataset is almost impossible.

Recently, Molchanov et al. [5] proposed a method for joint gesture spotting and gesture recognition using a zero or negative lag procedure through a recurrent three-dimensional convolution neural network (R3DCNN). This network is highly effective in recognizing weakly segmented gestures from multi-modal data.

In this paper, we propose an effective algorithm for both spotting and classification tasks by utilizing extracted 3D human and hand skeletal features.

#### **3. Proposed Algorithm**

In this section, we intently focus on the proposed method using two main modules: gesture spotting and gesture classification. For entire frames of continuous gesture video, the speed of hand and finger estimated from the extracted 3D human pose and 3D hand pose are utilized to segment continuous gesture. The isolated gesture segmented by gesture spotting module is classified using the proposed M-3D network with RGB, Optical flow, Depth, and 3D key joints information.

#### *3.1. Gesture Spotting*

The gesture spotting module is shown on the left of Figure 1. All frames of continuous gesture sequence are utilized to extract 3D human pose using the algorithm proposed in [23]. Through RGB hand ROI localized from 3D hand palm position *Jh(x,y,z)* when the hand palm stands still and over spine base joint, we use a 3D hand pose estimation algorithm to effectively extract the 3D position of the finger joints. From the extracted 3D human pose, the hand speed *vhand* is estimated using the movement distance of the hand joint between two consecutive frames.

• **3D human pose extraction:** From each RGB frame, we obtain a 3D human pose by using one of the state-of-the-art methods for 2D/3D human pose estimation in the wild-pose-hgreg-3d network. This network has been proposed by Zhou et al. [23] which provides the pre-trained model on the Human3.6M dataset [24]. This is the largest dataset providing both 2D, 3D annotations of human poses in 3.6 million RGB images. This network is a fast, simple, and accurate neural network based on 3D geometric constraints for weakly-supervised learning of 3D pose with 2D joint annotations extracted through the state-of-the-art of 2D pose estimation method, i.e., stacked hourglass network of Newell et al. [25]. In our proposed approach, we use this 3D human pose estimation network to extract the exact 3D hand joint information, which is effectively utilized for both gesture spotting and gesture recognition task.

Let *Jh*(*xhk, yhk, zhk*), *Jh*(*xhk*−1, *yhk*−1, *zhk*−1) be the 3D position of the hand joint at the *k*th frame, and (*k* − 1)th frame, respectively. The hand speed is estimated as

$$
\omega\_{\text{hundred}} = \mathbf{a} \cdot \sqrt{(\mathbf{x}\_{hk} - \mathbf{x}\_{hk-1})^2 + (y\_{hk} - y\_{hk-1})^2 + (z\_{hk} - z\_{hk-1})^2} \tag{1}
$$

where *α* is the frame rate.

The finger speed is estimated by the change in distance between the 3D position of fingertips of the thumb and the index finger in sequence frames. Let denote *Jft(xftk, yftk, zftk), Jfi(xink, yink, zink)* the 3D position fingertips of the thumb and the index finger at the *k*th frame, respectively. The distance between the two fingertips at the *k*th frame is given as

$$d\_{fk} = \sqrt{\left(\mathbf{x}\_{ftk} - \mathbf{x}\_{ink}\right)^2 + \left(y\_{ftk} - y\_{ink}\right)^2 + \left(z\_{ftk} - z\_{ink}\right)^2} \tag{2}$$

where *dfk* and *dfk*−1 represent the distances of the *k*th frame and previous frame, respectively, the finger speed *vfinger* is estimated as

$$w\_{finger} = a \cdot \left(d\_{fk} - d\_{fk-1}\right) \tag{3}$$

The function utilizes *vhand* and *vfinger* extracted from each frame:

$$
\upsilon\_k = \upsilon\_{hand} + \upsilon\_{finger} \tag{4}
$$

and is used as the input of the Bi-LSTM network to spot gestures from video streams, as shown in Figure 2. In our network, the Connectionist temporal classification [12] CTC loss is used to identify whether the sequence frames are in gesture frames or transition frames.

$$\begin{cases} \quad \dot{\boldsymbol{t}}\_{t} = \sigma(\boldsymbol{W\_{i}}[\boldsymbol{x}\_{t}, \boldsymbol{h}\_{t-1}] + \boldsymbol{b}\_{i} \\ \quad \boldsymbol{f}\_{t} = \sigma\Big(\boldsymbol{W\_{f}}[\boldsymbol{x}\_{t}, \boldsymbol{h}\_{t-1}] + \boldsymbol{b}\_{f} \\ \quad \boldsymbol{o}\_{t} = \sigma(\boldsymbol{W\_{o}}[\boldsymbol{x}\_{t}, \boldsymbol{h}\_{t-1}] + \boldsymbol{b}\_{o} \\ \quad \boldsymbol{\tilde{c}\_{t}} = \tanh(\boldsymbol{W\_{c}}[\boldsymbol{x}\_{t}, \boldsymbol{h}\_{t-1}] + \boldsymbol{b}\_{c}), \\ \quad \boldsymbol{c}\_{t} = \boldsymbol{f}\_{t} \* \boldsymbol{c}\_{t-1} + \boldsymbol{i}\_{t} \* \boldsymbol{\tilde{c}\_{t}}, \\ \quad \boldsymbol{h}\_{t} = \tanh(\boldsymbol{c\_{t}}) \* \boldsymbol{o}\_{t} \end{cases} (5)$$

where *i*, *f*, and *o* are the vectors of input, forget and output gate, respectively. *c*#*t* and *ct* are called the "candidate" hidden state and internal memory of the unit. *ht* represents the output hidden state. *σ*(.) is a sigmoid function while W and b are connected weights matrix and bias vectors, respectively.

**Figure 2.** Gesture segmentation with Bi\_LSTM and CTC loss.

• **Bi-LSTM network**: While the output of a single forward LSTM network depends only on previous input features, the Bi-LSTM network is known as an effective method for sequence labeling tasks, which is beneficial to both previous and future input features. Bi-LSTM can be considered as a stack of two LSTM layers, in which, a forward LSTM

layer utilizes the previous input features while the backward LSTM layer captures the future input features. The benefit of the fact that the Bi-LSTM network considers both previous and future input features is its effectiveness to classify the frame in sequence frame, gesture frame, or transition frame. The prediction error can be reduced by using Bi-LSTM instead of LSTM.

• **Connectionist temporal classification:** The Connectionist temporal classification CTC is known as the loss function which is highly effective in sequential label prediction problems. The proposed algorithm utilizes CTC to detect whether the sequence frames are in gesture frames or transition frames with input from a sequence of Soft-Max layer outputs.

#### *3.2. Gesture Classification*

The isolated gestures segmented by the present gesture spotting module are classified into individual gesture classes in the gesture recognition module. The proposed gesture recognition module is a multi-model network called the M-3D network. This model is based on a multi-channel network with three different data modalities, as shown on the right of Figure 1.

In our approach, from each frame of a video, we extract optical flow, 3D pose (hand joint, thumb tip, and index fingertip joint) information of multi-channel features input to the model. Optical flow is determined by two adjacent frames. There are some existing methods of optical flow extraction such as Farneback [28], MPEG flow [29], and Brox flow [30]. The quality motion information of optical flow clearly affects the performance of the gesture recognition model. Therefore, the Brox flow technique is applied to our approach as it has better quality performance compared to other optical flow extraction techniques.

While the key hand and finger joints positions are extracted by the 3D human pose and 3D hand pose extraction network presented in Section 3.1, we only focus on the two most important joints of thumb tip and index fingertip which can describe all gesture types. Our gesture classification algorithm is based on the combination of three 3D\_ResNet stream networks of RGB, Optical Flow, Depth channels with an LSTM network of 3D key joint features.

• **Three stream RGB, Optical Flow, and Depth 3D\_ResNet networks:** The 3D\_CNN framework is regarded as one of the best frameworks for spatiotemporal feature learning. The 3D\_ResNet network is an improved version of the residual 3D\_CNN framework based on ResNet [31] architecture. The effectiveness of 3D\_ResNet has been proved by remarkable performance in action video classification.

The single 3D\_ResNet is described in Figure 3. The 3D\_ResNet consists of a 3D convolutional layer and is followed by a batch normalization layer and rectified-linear unit layer. Each RGB and Optical Flow stream model is pre-trained on the largest action video classification dataset of the Sports-1M dataset [32]. Input videos are resampled into 16 frames-clips before being fed into the network. Let a resampled sequence of 16 frames RGB frames be *Vc =* {*xc*1, *xc*2, ... , *xc*16}, Optical Flow frames be *Vof* = {*xof*1, *xof*2, ... , *xof*16} and Depth frames be *Vd* = {*xd*1, *xd*2, ... , *xd*16} and operation function 3D\_ResNet network of RGB, Optical Flow and Depth modalities be Θc(.), Θof(.) and Θd(.), respectively. Hence, the prediction probability of two single networks for i classes is

$$P\_{\mathfrak{C}}\{p\_1, p\_2, \dots, p\_{16}|V\_{\mathfrak{C}}\} = \Theta\_{\mathfrak{C}}(V\_{\mathfrak{C}}) \tag{6}$$

$$P\_{of}\left\{p\_{1\prime}, p\_{2\prime}, \dots, p\_{16}\middle|V\_{of}\right\} = \Theta\_{of}\left(V\_{of}\right) \tag{7}$$

$$\Pr\{p\_1, p\_2, \dots, p\_{16} \vert V\_{\mathcal{D}}\} = \Theta\_{\mathcal{D}}(V\_{\mathcal{D}}) \tag{8}$$

where *pi* is the prediction probability of video belonging to the *i*th class.

• **LSTM network with 3D pose information:** In dynamic gesture recognition, temporal information learning plays a critical role in the performance of the model. In our approach, we utilize the temporal features by tracking the trajectory of the hand palm together with the specific thumb tip and index fingertip joint. LSTM framework is suitably proposed to learn the features for the gesture classification task. The parameters of our LSTM refer to the approach [33]. Input vectors from a sequence of the LSTM network frames is defined as: *Vj =* {*vj*1, *vj*2, ... , *vj*16} where: *vjk =* {*Jh*(*xhk, yhk, zhk*), *Jft*(*xftk, yftk, zftk*), *Jfi*(*xink, yink, zink*)} is a 9 × 1 vector which contains 3D position information of key joints at kth frame. The input of the LSTM network corresponds to the dimension of a single frame of sequences of 16 sampled frames in a gesture video that is a tensor for 1 × 9 numbers. The prediction probability output using LSTM with input *Vj* is

$$P\_L\{p\_{1\prime}, p\_{2\prime}, \dots, p\_{16} \vert V\_{\dot{\jmath}}\} = \Theta\_L\{V\_{\dot{\jmath}}\} \tag{9}$$

where Θ*L*(.) denotes the operation function of the LSTM network.

**Figure 3.** The overview of 3D\_ResNet architecture. This figure showed the number of feature map, kernel size of the 3D convolutional layer (3D Conv), batch normalization layer (Batch-Norm), and Rectified-Linear unit layer (ReLU).

• **Multi-modality fusion:** The results of the multiple different channel networks are fused in the final fusion layer to predict a gesture class. It is a fully connected layer where the number of output units is equal to the number of classes on the dataset. The output probability of each class is estimated by pre-trained last fusion layer with Θ*fusion*(.) operation function:

$$P\{p\_1, p\_2, \dots, p\_{16} \mid V\_{\varepsilon}\} = \Theta\_{f \text{sim}} \left\{ \begin{array}{ll} P\_{\varepsilon} \{p\_1, p\_2, \dots, p\_{16} \mid V\_{\varepsilon}\}, & P\_{of} \left\{p\_1, p\_2, \dots, p\_{16} \middle| V\_{of}\right\}, \\ & P\_{\mathcal{D}} \{p\_1, p\_2, \dots, p\_{16} \mid V\_{\mathcal{D}}\}, & P\_{\mathcal{L}} \{p\_1, p\_2, \dots, p\_{16} \middle| V\_{\mathcal{I}}\} \end{array} \right\} \tag{10}$$

The performance of the gesture recognition task is improved by combining the temporal information learning by LSTM network with spatiotemporal features learning by 3D\_ResNet that is proved through experimental results.

#### **4. Experiments and Results**

In this section, we describe the experiments that evaluate the performance of the proposed approach on three public datasets: 20BN\_Jester dataset [34], NVIDIA Dynamic Hand Gesture dataset [5], and Chalearn LAP ConGD dataset [35].

*4.1. Datasets*

This weakly segmented gesture video includes the preparation, nucleus, and transition frames of gesture.

• **Chalearn LAP ConGD dataset:** is a large dataset containing 47,933 gesture instances with 22,535 RGB-Depth videos for both continuous gesture spotting and gesture recognition task. The dataset includes 249 gestures performed by 21 different individuals. This dataset is further divided into three subsets: training set (14,314 videos), validation set (4179 videos), and test set (4042 videos).

The summary of the three datasets is shown in Table 1.

**Table 1.** Ablation studies on the ISBI 2016 and ISBI 2017 datasets.

#### *4.2. Training Process*

• **Network training for hand gesture spotting:** To train the Bi-LSTM network for segmentation of continuous gestures, we firstly use a pre-trained 3D human pose extraction network (on Human3.6M dataset) and a pre-trained 3D hand pose extraction network (on Stereo Hand Pose Tracking dataset) to extract the 3D position of key poses. The quality between human and hand pose extraction algorithms are demonstrated in Figure 4. Using those extracted input features for the network, we train the Bi\_LSTM network with the provided gesture segmentation labels by a training set of Chalearn LAP ConGD dataset.

**Figure 4.** (**a**) The 2D and 3D human pose estimation examples and (**b**) The 2D and 3D hand pose estimation examples.

Bi-LSTM network is trained with CTC loss for predicting the sequence of binary output values to classify whether the frame belongs to gesture frame or transition frame. In Bi\_LSTM, the input layer has 20 time-steps, the hidden layer has 50 memory units, and the last fully connected layer output has one binary value per time-step with a sigmoid active function. The efficient ADAM optimization algorithm [36] is applied to find the

optimal weight of the network. The spotting output of the Bi-LSTM network by a given speed input is displayed as in Figure 5.

• **Network training for hand gesture classification:** The single-stream network (pretrained on Sports-1M dataset) is separately fine-tuned on the huge dataset Chalearn LAP ConGD dataset. Each fine-tuned stream 3D\_CNN network weights is learned using ADAM optimization, learning rate with an initial value of 0.0001 reducing by half for every 10 epochs on 200 epochs. Ensemble modeling with 5 3D\_ResNet models is applied to increase the classification accuracy. The LSTM network parameters are selected through the observations of experimental results. The optimal LSTM model parameters are 3 memory blocks, and 256 LSTM Cells per memory block. The pretrained LSTM network is trained with a learning rate of 0.0001 on 1000 epochs. After pre-training of each streaming network, we retrain these networks with a specific dataset. Finally, we concatenate the prediction probability outputs of these trained models to train the weights of the last fusion fully connected layer for gesture classification. Besides training with the 3D\_ResNet framework, we also train with the 3D\_CNN framework to prove the effectiveness of the proposed algorithm.

**Figure 5.** Example of sequence frames segmentation by the Bi\_LSTM network. The blue line is the given speed input, and the red line is gesture spotting output (a value of 1.0 indicates the gesture frames).

Table 3 shows that our gesture recognition module obtained a positive result. The recognition performance is improved by using 3D data information of key joints. Moreover, the recognition performance of our method is among the top performers of existing approaches, with an accuracy of 95.6% on the 20BN-Jester Dataset and an accuracy of 82.4% on the NVIDIA Hand Gesture dataset.

• **Continuous hand gesture spotting classification:** To entirely evaluate our approach on continuous dynamic hand gesture spotting recognition, we apply the Jaccard index [3] for measuring the performance. For a given gesture video, the Jaccard index estimates the average relative overlap between the ground truth and the predicted sequences of frames. A sequence S is given by *i*th class gesture label and binary vector ground truth *Gs,i*, while the binary vector prediction for the *i*th class is denoted as *Ps,i*. The binary vector *Gs,i* and *Ps,i* are vectors with 1-values indicating the corresponding frames in which the *i*th gesture class is being performed. So, the Jaccard Index for the given sequence S is computed by the following formula of

$$J\_{s,i} = \frac{G\_{s,i} \parallel P\_{s,i}}{G\_{s,i} \parallel P\_{s,i}} \tag{11}$$

When *Gs,i* and *Ps,i* are empty vectors, the Jaccard Index *Js,i* is set as 0. For a given sequence S containing L number of true class labels *ls*, the Jaccard Index is estimated by the function:

$$J\_s = \frac{1}{l\_s} \sum\_{i=1}^{l} J\_{s,i} \tag{12}$$

For all testing sequence of n gestures: *s* = {*s*1, *s*2, ... , *sn*} the mean Jaccard Index *Js J* →*s J* →*s* is applied to evaluation as follows:

$$\overline{f\_s} = \frac{1}{n} \sum\_{j=1}^{n} f\_{s,\bar{j}} \tag{13}$$

The spotting-recognition performance comparison of our proposed approach to the existing methods by evaluation experiment on the test set of Chalearn LAP ConGD dataset is shown in Table 4.

**Table 2.** Gestures spotting performance comparison with different methods on NVIDIA Hand Gesture dataset. Bold values are highest indices.

**Table 3.** Gesture classification performance comparison of different methods on the 20BN\_Jester dataset and NVIDIA Hand Gesture dataset. Bold values are highest indices.

**Table 4.** The spotting-recognition performance comparison of our proposed approach to existing methods on the test set of the Chalearn LAP ConGD dataset. Bold values are highest indices.

From the results illustrated in Table 4, the mean Jaccard Index on the test set of the Chalearn LAP ConGD dataset shows that the proposed method achieves satisfactory performance on the dataset. By using 3D key joint features and multiples, the recognition performance is significantly enhanced.

#### **5. Discussions**

In Section 4.3, we have shown the effectiveness of our method on the three datasets. In terms of hand gesture spotting, we get the best results of both indexes on the NVIDIA Dynamic Hand Gesture dataset and Chalearn LAP ConGD dataset. The extraction of human pose and hand pose helps us track the hand movement more accurately and detect the beginning and the end of the sequence, avoiding the minor motion that could contaminate the following classification task. In the task of hand gesture classification, Table 3 presents the efficiency of the addition of modalities into our model on both the 20BN\_Jester dataset and the NVIDIA Dynamic Hand Gesture dataset. Different views of data are crucial to the performance of the hand gesture classification. Continuous gesture classification is more difficult when there are several kinds of gestures in one video, which means the capability of gesture spotting greatly influences the performance of gesture classification. In Table 4, we get the best results when doing both tasks on the Chalearn LAP ConGD dataset.

#### **6. Conclusions**

In this paper, we presented an effective approach for continuous dynamic hand gesture spotting recognition for RGB input data. The continuous gesture sequences are firstly segmented into separate gestures by utilizing the motion speed of key 3D poses as the input of the Bi-LSTM network. After that, each segmented gesture is defined in the gesture classification module using a multi-modal M-3D network. In this network, three 3D\_ResNet stream networks of RGB, Optical Flow, Depth data channel, and LSTM networks of 3D key pose features channel are effectively combined for gesture classification purposes. The results of the experiments conducted on the ChaLearn LAP ConGD Dataset, NVIDIA Hand Gesture dataset, and 20\_BN Jester dataset proved the effectiveness of our proposed method. In the future, we will try to include other different modalities to improve the performance. The tasks of gesture spotting and classification in this paper are performed separately into 2 steps. The upcoming plan is to do both tasks by one end-to-end model so that it is more practical in real-world problems.

**Author Contributions:** Conceptualization, G.-S.L. and N.-H.N.; methodology, N.-H.N.; writing review and editing, N.-H.N., T.-D.-T.P., and G.-S.L.; supervision, G.-S.L., S.-H.K., and H.-J.Y.; project administration, G.-S.L., S.-H.K., and H.-J.Y.; funding acquisition, G.-S.L., S.-H.K., and H.-J.Y. All authors have read and agreed to the published version of the manuscript.

**Funding:** This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIT). (NRF-2020R1A4A1019191) and also by the Bio & Medical Technology Development Program of the National Research Foundation (NRF)& funded by the Korean government (MSIT) (NRF-2019M3E5D1A02067961).

**Institutional Review Board Statement:** Not Applicable.

**Informed Consent Statement:** Not Applicable.

**Data Availability Statement:** Not Applicable.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


**Author details**

Bindhu Jayaprakash\* and Divya Nair Department of Biotechnology, Sri Shakthi Institute of Engineering and Technology, Coimbatore, India

\*Address all correspondence to<EMAIL_ADDRESS>
© 2022 The Author(s). Licensee IntechOpen. This chapter is distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/3.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

*Expression of Certain Cytokine Genes in Avian Cells Infected with Newcastle Disease Virus DOI: http://dx.doi.org/10.5772/intechopen.106921*


**3. Current advances in developing systemic biomarkers**

Systemic biomarkers approach is a relatively new area of biomedicine. However, over two decades of its existence [64], significant advances have been made. Great effort spent in this area not only improved analytical algorithms, but also underlined the importance of personalized approach. For example, many prognostic markers have been suggested for breast cancer in the literature, in particular for predicting survival. But data collected in separate studies led to striking discovery of the lack of overlap of the predictive genes in most of these studies. This emphasized the need of personalized approach even within tumor groups that share the same histomorphology [65]. The reason for discrepancies is debatable—divergent patterns of expression profiles might have been due to several analytical factors considered in the present chapter, but the solution holds the same: systemic biomarkers are only informative when patterns of pathways activation, rather than changes in individual genes expression, are analyzed. This idea led foundation to development of several analytical tools and panels. Our lab developed an NFE2L2/ AP-1 pathway-based systemic biomarker for assessing slight changes in physiological parameters of the human organism using peripheral blood leukocytes as the preferred sample type [66]. The same systemic approach utilizing another set of oxidative status markers was successfully used for unveiling features of uterine cervical incompetence patients [67]. Other labs also successfully apply pathway activation-based technologies in various field and other sample types, with special attention paid to fresh solid tumors samples and paraffin blocks [68]. Of note, Oncofinder technology [69, 70] and Oncotype DX assay [71] are among the most effective interactomics/multi-gene analysis-based tests in oncology.

In **Table 1**, some examples of suitable NFE2L2/AP-1 targets and complex markers are given along with their diagnostic properties (only area under the curve (AUC) is given, please see details in the cited publications).

As seen from **Table 1**, not only the markers may highly vary in nature, but they have different receiver operator characteristics. Notably, for each model to be studied, it is possible to choose or find a set of markers having extremely high AUCs that are hardly achievable using the traditional biomarker approaches.

**Acknowledgements**

**Conflict of interest**

corporate group.

present chapter.

**Author details**

**References**

1989;**86**:99-103

Viktor Fedoseev and Dmitriy Smirnov

DOI: 10.1016/j.cell.2009.12.028

Radiation Research. 2003;**159**:283-300

of Biological Chemistry. 2001;**276**:29251-29256

\*Address all correspondence to<EMAIL_ADDRESS>
Biomedical Innovations LLC, Rostov-on-Don, Russia

The chapter preparation was funded by Biomedical Innovations LLC, a part of Evolution

Oxidative Status Pathways: Systemic Biomarkers http://dx.doi.org/10.5772/intechopen.75543 105

The authors claim that there were no financial or other conflicts of interests related to the

[1] Kiel C, Yus E, Serrano L. Engineering signal transduction pathways. Cell. 2010;**140**:33-47.

[2] Keyse SM, Tyrrell RM. Heme oxygenase is the major 32-kDa stress protein induced in human skin fibroblasts by UVA radiation, hydrogen peroxide, and sodium arsenite. Proceedings of the National Academy of Sciences of the United States of America.

[3] Dent P, Yacoub A, Contessa J, Caron R, Amorino G, Valerie K, Hagan MP, Grant S, Stress S-UR. Radiation-induced activation of multiple intracellular signaling pathways.

[4] An JY, Zhou LL, Sun P, Pang HG, Li DD, Li Y, Zhang M, Song JN. Role of the AMPK signaling pathway in early brain injury after subarachnoid hemorrhage in rats. Acta

[5] Li WG. Miller FJ Jr, Zhang HJ, Spitz DR, Oberley LW, Weintraub NL. H(2)O(2)-induced O(2) production by a non-phagocytic NAD(P)H oxidase causes oxidant injury. Journal

Neurochirurgica. 2015;**157**:781-792. DOI: 10.1007/s00701-015-2370-3

Peter Zolotukhin\*, Viktor Chmykhalo, Anna Belanova, Alexander Dybushkin,

**Table 1.** The NFE2L2/AP-1 pathway functioning markers used in two in vitro and in vivo studies, with AUC in descending order within each study.


Charcoal produced at intermediate pyrolysis pressure (10 bar) showed a smaller surface area than charcoal generated at low pressure (5 bar), respectively high pyrolysis pressure (20 bar) [263]. Yang et al. reported that the surface area decreased with increasing heating rate [35] and was generally lower for herbaceous charcoal than for woody charcoal [32]. The results indicate that the release and secondary pyrolysis reactions of volatile matter has a high impact on the surface area of charcoal. Fossil fuel reductants, such as metallurgical coke, bio-coke and petroleum coke, exhibit a surface area of less than 3 m2 g−1 [21,63,267]. Metallurgical coke and bio-cokes are produced at temperatures above 1100 ◦C, while petroleum coke is the solidified residue from crude oil distillation. Differences in the accessibility were observed for the surface characterization in N2 and CO2. Surface area from oak and pine charcoal increases from less than 3 m2 g−1 at 400 ◦C to 220–280 m2 g−1 at 650 ◦C for measurements in N2, respectively from 250 to 530 m2 g−1 for oak and 360 to 650 m2 g−1 for pine in

CO2 [268]. Thus, the accessibility of gas species into the pore volume can also affect the reactivity in different atmospheres.

#### *4.4. Mechanical Properties*

Mechanical properties of charcoal are essential to avoid generation of fine particles by transport, handling and feeding into the SAF. The furnace operation and off-gas cleaning system are significantly affected by the particle size of original feedstock and charcoal [18]. In general, charcoal is more fragile than fossil fuels samples due to its high porosity and lower carbonization degree [253]. Global charcoal transport results in the generation of 5–20% fines [159,269–271] due to the loading and unloading of the charcoal [271]. This material is not directly suitable for a usage in SAF, since a particle size of 5–40 mm is required for ferroalloy industry to ensure a good gas permeability [29,239]. In particular, the silicomanganese industry uses fossil fuel reductants with a particle size ranging from 5 to 20 mm [29]. Thus, particles less than 5 mm are considered as fines and should be avoided.

Standard tests for measuring the mechanical properties are drum tests, such as ASTM D 294-64, ISO 556 and DIN 5171 for coke, and DIN EN ISO 17831-1 for pellets and briquettes. In addition, mechanical fragmentation is investigated by the drop shatter test for coal according to ASTM D440. Hot strength of metallurgical coke is investigated by the coke strength after reaction (CSR). The combination of hot strength and CO2 reactivity are important parameters to evaluate the quality of reductants in the burden [37], especially for blast furnaces. However, most of the standards are not applicable for charcoal and have therefore been modified by the researchers [36,37].

While a very high compression strength is required in blast furnaces, mechanical strength is of less importance in SAF due to the low height of the burden [40,239]. Compressive strength of charcoal is 10–80 kg cm−2, about half of that of metallurgical coke (130–160 kg cm−2) [36,201]. Charcoal produced from hardwoods have generally a greater mechanical strength than charcoal from softwood or herbaceous biomass [152] and the mechanical strength is 3 to 4 times greater in the growing direction of wood fibers compared to its perpendicular direction [36]. The compression strength slightly increased after the sample was heat treated to elevated temperatures [201,272], indicating improved mechanical properties inside the SAF. Carbonized wood is about 28% stronger than its biomass precursor [273].

#### *4.5. Charcoal Compaction*

Charcoal produced at elevated temperatures has a lower tendency to agglomerate compared to raw biomass and requires the addition of a binder for compaction [123]. Charcoal fines can be compacted by pelletization and briquetting to increase bulk density and mechanical stability for the use in metallurgical industry [124,172,274]. Coal tar, coal tar pitch and bio-oil pitch have been used as binder for special purposes over decades [271]. Recent studies have confirmed that the mechanical properties can be improved for charcoal pellets and briquettes by the addition of bio-oil and bio-oil pitch and a second heat treatment [63,150,275]. The mechanical durability of the pellets increased to greater than 90% [150], similar to that of heat treated hydrochar pellets [129]. Pellets compacted at 116 MPa with an oil concentration of 34% provided optimal pelletizing conditions [180]. A slightly better mechanical durability was achieved by adding water and molasses at a compaction pressure of 100 MPa [172]. One of the major disadvantages of molasses is the addition of mineral matter (especially potassium) to charcoal pellets and briquettes. Other important parameter are particle size and moisture content, and the glass transition temperature for lignocellulosic binder [124]. Drop strength tests approved the results from mechanical durability test for charcoal pellets [172].

The compressive strength of charcoal pellets increased from less than 1 MPa to greater than 2 MPa after the secondary heat treatment [150]. A similar compressive strength was reported for charcoal briquettes using bio-oil binder [275]. The mechanical compression resistance of green pellets produced with molasses as a binder increased from 1 MPa to 2.6 MPa after storage for 12 days [172]. Similar values were found for fossil fuel briquettes compacted at 56 kPa [199], in which a higher

compaction pressure resulted in improved mechanical properties and hardwood lignin developed twice the compressive strength than softwood lignin. Lignin as a binder was improved by adding tannic acid, in which the mechanical strength was also obtained as hot strength for temperatures up to 1400 ◦C [199]. A compressive strength of 4.2 MPa was reported for briquettes using starch binder [275], which was similar to that of charcoal pellets using bio-oil binder [150]. To ensure mechanical strength at higher temperatures, silicon metal can be added to the charcoal fines, forming a network of silicon carbide nanowires between the carbon matrix [276]. The carbide nanowires improved the resistance to compaction from 1 MPa to 3–4 MPa at 1200–1400 ◦C [199].

Pellets started to shrink after the primary heat treatment temperature was surpassed [142]. This additional size reduction must be considered if the particle size of charcoal pellets is close to the minimum particle size requirements. Pellets from fast pyrolysis and slow pyrolysis differ by its aromaticity and volatile matter content [172], leading to differences in a shrinkage at higher temperature. If lignin is selected as binder, a secondary heat treatment temperature above 450 ◦C should be chosen to form polyaromatic structures between the particles [277]. The additional heat treatment at elevated temperature also reduces the risk of self-heating under transport and storage [274] and decreases CO2 reactivity [142]. In summary, the quality of pellets and briquettes is highly improved by the second heat treatment.

#### *4.6. Electrical Properties*

The electrical conductivity is one of the most important properties of carbon reductants in SAF and EAF [239,253,278]. The electrical conductivity of charcoal is affected by the heat treatment temperature (carbonization degree), density of the packed bed, particle size and volume fraction of carbon, as well as operating temperature and pressure inside the furnace [1,201,253,279,280]. Metallurgical coke used in SAF is expected to have an electrical resistivity of 7–10 mΩ m at room temperature and 1–10 mΩ m at 1400 ◦C for a particle size of 5–20 mm [37,239,280]. Charcoal on the other hand has an electrical resistivity above 106 Ω m at room temperature, decreasing to 14–23 mΩ m at 1000 ◦C and 9–18 mΩ m at 1400–1600 ◦C for 5–35 mm respectively 5–10 mm particles [37,201]. The electrical resistivity of charcoal particles from a heat treatment at 950 ◦C was similar to that of fossil fuel coal char ranging from 1.7 to 3.4 mΩ m [281].

The electrical resistivity of packed carbon beds is higher than that of particles due to larger contact resistance between particles [279,282]. Electrical resistivity of metallurgical coke decreased from 10–15 mΩ m at 1000 ◦C to 5–8 mΩ m at 1600 ◦C [37,201]. Packed beds of charcoal particles larger than 2 mm showed an electrical resistivity of 50 mΩ m at 1100 ◦C and 15–20 mΩ m at 1600 ◦C [37,201]. The electrical resistivity of metallurgical coke decreased by 50% when the particle size increased from 5–10 mm to 15–20 mm [280]. However, the previous results have also shown that only minimal differences were observed for small and large sizes of charcoal particles [201]. The compaction pressure of the burden has a large impact on the electrical resistivity. An increased compaction pressure decreases the air gap between particles and increases the number of contact points between the particles, resulting in a decreased electrical resistivity [18,201,279,283].

The loss in ohmic resistance can be attributed to the removal of oxygen groups by carbonization from the carbon matrix [252,284], especially at temperatures less than 950 ◦C. The high oxygen content and disordered carbon structure act as insulators in the carbon matrix of biomass, charcoal and coal [1,285–287], in which a decrease in electrical resistivity of 5 orders of magnitude was observed after the release of volatile matter [1,288]. Therefore, electrical resistivity of carbon reductants can be correlated to the ordering of the carbon matrix and its reactivity [281]. The higher electrical resistivity of charcoal inhibits the conduction of current through the burden inside the SAF [288,289], where ≈5–15% of current is conducted through [18]. In addition, the higher electrical resistivity of the charcoal enables a lower electrode tip position, which improves heat distribution in the lower part of SAF [280].

#### *4.7. Gas Reactivity*

Gas reactivity of carbon reductants is an important property for the burden in SAF. About 30% of the CO2 emission from ferro-managnese and silicomanganese production are related to the Boudouard reaction (C(s) + CO2 - 2 CO) [290], indicating the importance of a low gas reactivity towards CO2. A high reaction rate will increase the coke and power consumption and is therefore undesirable in silicomanganese and ferromanganese production [36]. The CO2 reactivity of metallurgical coke is investigated according to coke reactivity index (CRI), where the mass loss is measured by difference after a 2 h heat treatment (1100 ◦C) in pure CO2. The standard can not directly be adopted for charcoal samples due to its high gas reactivity. By AC-method, charcoals reactivity can be 30 to 40 times larger compared to metallurgical coke [36]. Many researchers investigate the CO2 reactivity by thermogravimetric analysis at lower temperature, lower CO2 concentration or by non-isothermal analysis [37,63,204,291,292]. In addition, knowledge of the CO2 reactivity can be adopted from fixed bed biomass gasification for syngas production [293].

The CO2 reactivities of metallurgical coke and charcoal samples produced at 1060 ◦C were compared. Carbon conversion of metallurgical coke was in the order (0.2–0.5)·10−2%C s−1, and thus, about 5 to 16 times lower than that for industrial and laboratory charcoal that ranged from (2.1–3.2)·10−2%C s−1) [37]. Similar results were obtained by other researchers [105,204], in which charcoals from forest residues and herbaceous biomass generally have a higher reactivity than those from stem wood [204]. CO2 reactivity of charcoal samples post treated at temperatures larger than 1600 ◦C approached that of fossil fuel reductants [62,99]. Despite the high porosity and surface area of charcoal, CO2 reactivity of charcoal samples decreased by about 20–50% when the particle size from 60 μm to 1–2 mm [294,295]. Previous studies have shown that up to 70% of the carbon conversion depends on the external surface area and the potassium content in original feedstock [291]. The higher reactivity of carbon fines may improve the stability of SAF by the consumption of small particles in the burden, resulting in an improved gas permeability of the bed.

Alkali and alkali earth metals are stated as catalysts for the Boudouard reaction [158,204,291]. The catalytic index is shown as the quotient of the sum of catalytic elements (K, Ca, Mg, Na, Fe) to inhibiting elements (Si, Al) [291]. Bio-based reductants with a low mineral matter content showed a similar reactivity compared to the fossil coke [20]. However, potassium impregnated coke was more reactive than low ash charcoal, but less reactive than potassium impregnated charcoal [36]. These results indicate a dominating role of potassium on the CO2 reactivity, confirming the previous results [97]. In addition, alkali and alkali earth metals can recirculate in the burden, resulting in an accumulation of potassium in the SAF [36]. The CO2 reactivity is also highly affected by CO2 partial pressure [200,296–299]. In addition, the high concentration of CO is desirable for ferroalloy production to pre-reduce higher metal-oxides in the burden and as a value-added product gas for other applications [300,301]. Beside the Boudouard reaction, charcoal also reacts with SiO gas in silicon and ferrosilicon production. The SiO reactivity is determined by passing hot SiO gas (13.5% SiO, 4.5% CO) through a carbon bed according to the improved SINTEF procedure, which was developed by Tuset and Raaness in the 1970's and described elsewhere [302]. The SiO gas will react with the carbon reductant according to: SiO(g) + 2 C(s) - SiC(s) + CO(g), thereby forming a solid silicon carbide layer at the surface. Therefore, it can be assumed that the SiO reactivity is mainly depending on the porosity and accessible surface area of the carbon material, eventually blocking the reaction surface and decreasing SiO reactivity over time [94]. The general trend is SiO reactivity of charcoal ≥ coal ≥ metallurgical coke ≥ petcoke [94,303,304]. It is also known that about 80% of the silicon from the gas phase can be recovered in the ferro-silicon production, while 20% of the SiO gas is discharged by the off-gases [239]. Therefore, woodchips and charcoal can improve the gas permeability of the burden, prevent the crusting of the charge and increase the recovery of SiO gas [253,305].

#### *4.8. Slag Reactivity*

Reduction of metal-oxide to the pure metals takes place in the liquid state by carbon reductants in the lower part of SAF, for example by the reaction: MnO + C - Mn + CO. Carbon can be provided as solid carbon at the contact point between the slag and the "coke" bed, and by the dissolution of carbon into the liquid metal. The sessile drop wettability test [306] and thermogravimetric analysis (TGA) at 1600 ◦C [37] are techniques to evaluate the slag-carbon reactivity in ferroalloy industries. Thus, slag reactivity can be characterized by its dissolution and by the reaction kinetics [253]. In the former, the slag reactivity is depending on the dissolution rate of the carbon material into the liquid metal phase. Alkali metals in charcoal can decrease the carbon solubility into the liquid metal. It is known from carbon dissolution into liquid iron that S, P, Si, Al, Ni and Co decrease the solubility of carbon, whereas Mn and Cr slightly increase it [253]. Charcoal and fossil coke can be equally used as reductants in silicomanganese production [36], in which the dissolution rate constant is significantly improved by increasing heat treatment temperature [37,253] due to the increased degree of crystallinity of the carbon matrix [94]. For example, the dissolution rate constants of coal can vary in the range 0.0011 to 0.0036 s−1 [253].

The kinetics of metal-oxides reduction can be investigated by using TGA coupled with an infrared analyzer at temperatures above 1500 ◦C [307,308]. A high graphitization degree of the carbon matrix is beneficial for the high reaction rate. However, reductants with a low degree of graphitization and large occupying volume fraction tend to have a higher reactivity towards manganese ores due to their large surface area [278]. Coke has a faster slag reactivity than charcoal and much faster than graphite, whereas the highest reactivity of charcoal is observed in alkali lean charcoal samples [309]. However, the kinetics for slag and MnO reduction can change by the presence of metal phases, such as reduced FeO and SiO2 [253,309]. Silicon and ferrosilicon production can be accounted as slag free [30], and the main reduction reaction is given by the SiO-reactivity as discussed above. Thus, highest quartz-carbon reactivity can be observed for charcoal and coal [304]. The different tendencies in reactivity of gas-carbon and slag-carbon inhibit a correlation between the two properties [306].

#### **5. Available Charcoal Technologies**

Charcoal has been produced batch-wise in different kiln technologies since millenia, and some of the old inefficient technologies are still used in developing and newly industrializing countries [8]. In general, charcoal production can be divided into kiln technologies where the heat is provided directly by the partial combustion of additional feedstock, respectively [25]. A schematic of the different heat supply system for charcoal production is shown in Figure 2, whereas the production capacity and production rate are summarized in Table 7. In general, maximum heat treatment temperature for charcoal production in kilns and retorts is between 450 and 650 ◦C [68,161,220].

The pits and earthmound kilns are the simplest technologies for the charcoal production. The space capacity of the pits are in the range of 4 to 30 m3 of fuel wood, while earthmound kilns can be enlarged up to a capacity of 150 m3 [25]. Earthmound kilns have been improved in the second half of the 19th century by adding a chimney, and provided large amounts of charcoal for iron smelting industry [25]. The additional chimney enabled a precise control of the air supply by ensuring optimum draught conditions, and resulting in a greater charcoal yield with better properties [25,310]. The chimney can also act as an air cooler to condense heavy hydrocarbons, acids and water, which are value-added by-products from pyrolysis process and improve the control of the kiln. Chidumayo estimated that it is unlikely to futher increase the charcoal production rate of 23.3% in earthmound kilns [311]. Brick kilns have been used and improved over the last decades. The Missouri kilns provide a volume capacity of 180–800 m3 and a charcoal yield of 20–30% [160,161], about twice than that of classical earthmound kilns. The cycle time of these kilns is typically in the range from 7 to more than 30 days [11,160,226]. Largest kilns have a capacity of 2000 m3 and can produce 10,000 tons of charcoal per year [161].

Most of the charcoal in the EU is produced in steel retorts [161]. Biomass carbonization is thereby carried out in continuously operated processes, such as the Lambiotte retort, which is used since 1942 with a production capacity of 2000–9000 tons of charcoal per year [68]. Heat treatment temperature in Lambiotte retort is 600 ◦C in the carbonization zone [68]. Other technologies, such as the Carbon engineering technology are operated at 500 ◦C that is similar to the maximum temperature in kilns [161,220]. A horizontal retort (also called Twin-retort) has been developed in the 1990s by Norit B.V., in which multiple retorts can be combined to increase production rate and reduce emissions [6]. Similar technologies are the VMR systems with two retorts and one central combustion chamber and a production rate of 6000–7000 tons per year [226]. Other retorts are Reichert retort, Lurgi/Degussa process, Nichols-Herreshoff Process and the rotative furnace [8,25]. Carbonex is the most recent process, combining the pyrolysis unit with a 1.4 MW power production [161]. The largest industrial charcoal plants produce about 25,000–27,000 tons of charcoal annually [207,208].

**Figure 2.** Possible heating systems in charcoal production (after [25,29]).

#### *5.1. Economics*

Charcoal markets are established in Africa, Asia and South-America, where charcoal is the primary fuel source to urban and rural households. The largest wood charcoal markets in 2018 were Brazil, Ethiopia and Zambia, with a total market size of about 6.7 billion US dollars [312]. This charcoal was often produced by earthmound kilns at a low efficiency, but also at low costs [311]. Due to the low capital expenditures (CapEx) and operating expenses (OpEx), the economics of such charcoal production is mainly driven by the charcoal market prices and capacity factor for charcoal production [6]. The charcoal production in the EU countries is always highly affected by the investment costs, OpEx and biomass price, such as energy saving and production efficiency [166]. Charcoal price of 250 Euro per ton of

charcoal in 2002 was required to cover investment costs of about 500,000 Euro [6]. However, the cost of the biomass feedstock is one of the major obstacles for charcoal or bio-oil production [172].

Sweden, Norway and Finland have large unused potential of forest areas available for wood production, e.g., 19.6 million ha in Finland [313] or about 10–15 million m3 in Norway [200]. Cornifer roundwood, such as spruce and pine, are the main wood species in the northern countries, accounting for more than 75% of the annual harvest [313,314]. Some sawlogs may not provide the strength characteristics for a construction industry and can be considered as a possible feedstock for charcoal production [313]. There is a deficit of biomass mobilization and only limited integration of biomass resources in other industrial sectors [26], which is also valid for global roundwood markets with a total global trade volume of 135 million m3 (7% of industrial roundwood production) [313]. The limited market integration can lead to the fluctuations in prices and can also inhibit investment in new technologies or methods to produce charcoal, whereas the metallurgical industry can be a large scale customer for renewable reductants over the long term.

Bio-refineries can improve the economics of charcoal production routes by enabling integrated value-added products at reduced fuel prices, and can be a possible key for future commercialization [315]. Bio-fuel production, which is well integrated in Europe, can establish the first stage of a bio-refinery [60], and the solid residues from the emerging bio-fuel markets can be used as feedstock for metallurgical grade charcoal production. The principles of bio-refinery can improve the overall economics by utilizing the complete feedstock or minimizing waste stream generation [316]. Charcoal can be extracted from bio-crude production, which would result in an increased number of decentralized pyrolysis plants to cover the bio-fuel product stream [317]. Bio-oil can be processed and specific compounds can be separated and used as value-added products [178]. For example, a natural smoke was extracted and commercialized by Ensyn as a food additive, whereas acetic acid can be used by chemical industry to produce vinyl acetate or ethyl acetate [178].

Charcoal production in developing countries is often carried out by unskilled operators in pits and earthmound kilns without a licence [221]. In some countries more than 75% of energy demand are covered by firewood and charcoal [318,319]. However, the income is especially important for households with loss agricultural capacity and limited stocks [318] and can account for more than 2% of GDP of the country [221]. Although the cost share of licensing is very low, sustainable wood sourcing requires the adequate governmental regulations to reflect the sustainability of a sustainable secondary forestry [7]. Certification of charcoal by Programme for the Endorsement of Forest Certification (PEFC), Forest Stewardship Council (FSC) or European Biochar Certificate (EBC) can help to distinguish between sustainable charcoal produced from secondary forestry and charcoal produced from wood made from primary forestry. Modern technologies can also increase the conversion efficiency and reduce the risk of local pollution, while long term contracts can ensure a stable price range for local charcoal producers and the ferroalloy industry.

#### **6. Discussion**

Charcoal has been used in the metallurgical industry over millenia and is partly used in the ferroalloy industry, such as silicon and silicomanganese production in Brazil. Typical reductants used for the silicon production are coal, petroleum coke and charcoal, while woodchips are used to ensure a good gas permeability in the burden [23]. Thus, fossil fuel reductants and renewable reductants are considered as carbon sources in silicon metal production [320]. Woodchips and charcoal may fully replace fossil-based reductants when their properties are comparable with the requirements for the use in open furnaces, e.g., reactivity, mechanical abrasion and mechanical strength. Minimum requirements vary by ferroalloy production and furnace type, in which reactivity toward CO2 and dissolution rate into the metal are important parameters for ferromanganese and ferrochromium production, whereas SiO reactivity is important for ferrosilicon and ferrochromium reduction [94]. The high volatile matter content of classical charcoal is larger than 10 wt.% resulting in off-gas cleaning issues in closed hearth furnaces during reduction of manganese ferroalloys (both FeMn and SiMn).

Charcoal fines can cause two types of problem. Charcoal fines can reduce the gas permeability in the burden, resulting in an unstable operation of the furnace. Another challenge is that charcoal fines can be carried over into the gas cleaning system by the generated gases. Transport and handling can result in a fines generation up to 20%, which are not suitable in SAF. Charcoal pellets mixed with the bio-oil binder showed similar mechanical properties without an impact on the CO2 reactivity to that of fossil-based coke. The use of organic binders i.e., molasses or starch resulted also in high mechanical stable pellets and briquettes [275]. However, the high ash content in molasses i.e., K, Na, Ca can interact with the organic compounds in other organic binders. The disadvantage of charcoal pelletization is the requirement to comminute particles. Briquettes can be produced from fines without additional milling and with satisfactory mechanical properties for the ferroalloy industry. The secondary heat treatment is required to ensure the high mechanical stability of charcoal-based pellets or briquettes.

The large volatile matter content and high CO2 reactivity of charcoal may cause problems in the burden of SAF. The structural and chemical properties, mineral matter composition of the charcoal can be adjusted using post-treatment processes [291]. While volatile matter content decreases with increasing pyrolysis temperature, the ash content increases inversely proportional to the release of volatile matter. The increased ash content in charcoal may inhibit the use of alkali rich feedstock i.e., wheat straw, rice husks, bark, etc. In addition, the high alkali metal content of charcoal may cause problems in the upper part of the SAF where alkali metals tend to accumulate. In addition, different post-treatment processes, such as acid leaching and high temperature treatment, have shown that the ash content and reactivity of charcoal can be optimized approaching properties of fossil fuel reductants.

Currently, most of the charcoal is produced in developing countries and is locally used for heating and cooking. An increased global demand can be covered in a certain range, but a sustainable production may be limited due to the availability of energy efficient retorts or kilns. Earthmound kiln and pits can emit harmful emissions and pollute local environment by unskilled operators, such as local farmers involved in the unlicensed charcoal production. Deforestation and inefficient charcoal production can finally result in the net GHG emissions and should then be avoided as a substitute to fossil-based reductants. Certificated charcoal or charcoal from certified producers can reduce the risk of charcoal from primary forestry, e.g., by Forest Stewardship Council (FSC) or European Biochar Certificate (EBC). In the long term, modern charcoal retorts can produce high quality charcoal at high conversion efficiency, ensuring an income for farmers and operators at a stable price.

Charcoal production in Europe can ensure a sustainable biomass production and efficient charcoal manufacturing. However, the lack of market integration and fluctuation of feedstock prices may inhibit investments in new technologies. Bio-refineries can combine the production of chemicals, fuels and reductants. Recent research has shown that properties of charcoal were not negatively affected by recovering value-added compounds before pyrolysis, for example waxes, steroids or fatty acids. The liquid by-product from pyrolysis can be used as a binder to enlarge the charcoal yield with the limited influence on the CO2 reactivity.

#### **7. Summary**

Charcoal can be considered as a renewable reducing agent for the ferroalloy industry. Retorts operated at 550–600 ◦C produce a charcoal with a fixed carbon content greater than 85%, in which pyrolysis by-products are utilized to provide the thermal heat for the process. Charcoal produced under these conditions can directly be used in open hearth furnaces for the ferrosilicon and silicon production. The high SiO reactivity of charcoal makes it a superior reductant compared to coal, semi-coke or coke. However, the mechanical properties of charcoal are generally inferior to fossil-based reductants, resulting in up to 20% losses of fines by handling and transport. In addition, the compressive strength of charcoal is half compared to metallurgical coke, but still sufficient to withstand the mechanical load by the burden in SAF.

Charcoal fines can be densified through pelletization or briquetting using organic binder such as bio-oil, starch or lignin to reduce waste streams. It was shown that the mechanical properties of densified charcoal are improved by a second heat treatment and would also withstand the load of the burden in SAF. The organic binders i.e., bio-oil are preferred over the inorganic binders due to the impact of alkali matter content on the reactivity and mechanical strength of charcoal samples. The high potassium content in charcoal may be a hurdle, since it catalyzes the Boudouard reaction. Acid leaching of biomass feedstock or charcoal can reduce the potassium content, and in combination with a high temperature treatment, concurrently adjust CO2 reactivity that approaches properties of fossil-based reductants. A secondary heat treatment at temperatures above 1100 ◦C is required to reduce the volatile matter content to below 1.5%.

Charcoal production in developing countries is only sustainable in a certain range. An additional charcoal demand is expected to increase local prices, possibly leading to a larger charcoal production using primary forestry. This can result in a short-term deforestation and charcoal production by inefficient technologies. Certification such as European Biochar Certificate (EBC) can reduce the risk of primary forestry and inefficient charcoal production. Using certified charcoal can substantially contribute to the reduction of anthropogenic CO2 emissions from ferroalloy industry.

Harvesting cycles of 10 years can provide a metallurgical grade charcoal with the properties approaching that of currently used fossil-based reductants. Large potential of wood is available in Norway, Sweden and Finnland. However, the high CapEx and OpEx for classical charcoal production inhibit local charcoal development. Bio-refineries can provide high value-added chemicals for the energy and chemical sector with the concurrent production of charcoal used by metallurgical industries.

**Author Contributions:** Conceptualization, G.R.S., A.T. and M.T.; methodology, G.R.S.; validation, G.R.S., A.T. and M.T.; writing—original draft preparation, G.R.S.; writing—review and editing, G.R.S., A.T. and M.T.; visualization, G.R.S.; supervision, M.T.; project administration, M.T.; funding acquisition, M.T. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by Research Council of Norway grant number 280968. The APC was funded by Norwegian University of Science and Technology.

**Acknowledgments:** The authors gratefully acknowledge the financial support from Research Council of Norway (Project Code: 280968) and our industrial partners Elkem ASA, TiZir Titanium & Iron AS, Eramet Norway AS, Finnfjord AS and Wacker Chemicals Norway AS for financing *KPN Reduced CO*2 *emissions in metal production* project.

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **Abbreviations**

The following abbreviations are used in this manuscript.

#### **References**

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

MDPI St. Alban-Anlage 66 4052 Basel Switzerland Tel. +41 61 683 77 34 Fax +41 61 302 89 18 www.mdpi.com

*Processes* Editorial Office E-mail<EMAIL_ADDRESS>www.mdpi.com/journal/processes

MDPI St. Alban-Anlage 66 4052 Basel Switzerland

Tel: +41 61 683 77 34 Fax: +41 61 302 89 18

www.mdpi.com