*2.3. IMU*

Two wireless MTw IMU sensors (Xsens, Enschede, The Netherlands) are used, which are placed on the users' body segments, (weight: 10 g, size: 47 × 30 × 13 mm). Each MTw sensor comprises a 3D gyroscope, a 3D accelerometer and a 3D magnetometer. IMUs sends its data to a station (Awinda, Xsens) via a proprietary wireless communication protocol at 100 Hz of sampling frequency. The orientation data,

in quaternion format, was estimated with a customized Kalman filter developed by the manufacturer (XKF3-hm) [32].

We implemented a method using two IMU units to estimate the elbow flexion-extension angle. The IMU on the proximal body limb (i.e., upper arm) was used as a reference, hereafter referred to as *S*1. The second IMU, which was located on the distal body limb, will be referred as *S*2. The algorithm uses five coordinate systems for its operation. *S*1 and *S*2 are the local coordinate system of the IMU sensors, these are embedded in each sensor. The technical-anatomical coordinate systems (*B*1 and *B*2) correspond to the body limbs' orientation [33]. Finally, the global coordinate system, *G*, is a frame formed by gravity acceleration vector and Earth's magnetic north. The reference IMU sensor (*S*1) was carefully aligned with the body limb, in such a way, during calibration moment, with a neutral standing posture and after applying a gravity alignment, *B*1 (*B*2) coordinate system is estimated from *S*1 (*S*2).

The algorithm aims the estimation of the angle between the orientation of the proximal and distal limb. Therefore, the limbs' orientations should be projected to a common coordinate system (i.e., global coordinate system). This is done through the orientation quaternions, *G***q***B*1 (*t*) and *G***q***B*2 (*t*). The estimation process presupposes that *x*-axis of both technical-anatomical coordinate system are aligned with gravity vector at the calibration moment in a straight neutral posture, as proposed in previous works [33–35]. Besides the elbow joint is simplified as a 1-DOF hinge joint. The flexion-extension angle (*α*) is calculated following the steps below:

$$\mathbf{^Gq}\_{B\_1}(0) = \mathbf{q}\_c \otimes \mathbf{^Gq}\_{\mathbf{^Gq}}(0) \tag{2}$$

$$\mathbf{^Gq}\_{B\_1}(0) = \mathbf{^Gq}\_{B\_2}(0) \tag{3}$$

where ⊗ denotes the Hamilton product. Please note that due to the assumptions mentioned above, at the calibration moment, *G***q***B*1 (0) and *G***q***B*2 (0) are equals. Which means that the initial angle in a straight neutral posture is zero.

• To calculate sensor orientation *S*1 (*S*2) with respect to its associated body segment *B*1 (*B*2) using Equation (4). Please note that *B*1**q***S*1 and *B*2**q***S*2 are constant at all time instants.

$$\mathbf{^{B\_1}q\_{S\_1}} = \left(^{G}\mathbf{q\_{B\_1}}(0)\right)^{-1} \otimes ^{G}\mathbf{q\_{S\_1}}(0) \tag{4}$$

• To estimate the body segments' orientation *G***q***B*1 (*t*) and *G***q***B*2 (*t*) using Equation (5). Please note that Equations (4) and (5) are expressed in terms of *B*1 and *S*1, but they are also applicable to *B*2 and *S*2.

$$\mathbf{^Gq}\_{B\_1}(t) = \mathbf{^Gq}\_{S\_1}(t) \otimes \left( {^{B\_1}\mathbf{q}}\_{S\_1} \right)^{-1} \tag{5}$$

• To calculate the relative orientation between *G***q***B*1 (*t*) and *G***q***B*2 (*t*). In that way, *B*1**q***B*2 (*t*) is then decomposed into 'ZXY' Euler angles. The rotation about *z*-axis represents the elbow flexion-extension angle, *α*, consistent with the ISB recommendations [16,36].

$$\mathbf{q}^{B\_1}\mathbf{q}\_{B\_2}(t) = \left(^G\mathbf{q}\_{B\_1}(t)\right)^{-1} \otimes ^G\mathbf{q}\_{B\_2}(t) \tag{6}$$

#### *2.4. Sensors Characterization*

To validate the measurements of the camera-based and IMU-based systems a goniometer as a standard reference is used. This reference has two adjustable lever locks, which are positioned to limit the flexion-extension motion between 20◦ (lower bound) and 90◦ (upper bound). The goniometer was placed and aligned with the elbow joint of the subject M1, then, this was asked to perform flexion-extension movements on the sagittal plane in such a way to reach both locks. Lastly, the data was acquired by the camera-based system and the IMUs. The POF curvature sensor is characterized using the procedure mentioned in Section 2.2.

#### *2.5. Experimental Protocol*

Eleven participants without motor impairments were enrolled in this study. Six females, referred as F1, F2, F3, F4, F5, and F6, age: 27.3 ± 4.9 years old, corporal weight 56.8 ± 16.3 kg, and five males, referred as M1, M2, M3, M4, and M5, age: 27.4 ± 3.3 years old, corporal weight 70.2 ± 3.8 kg, as shown in Table 1. This research was approved by the Ethical Committee of UFES (Research Project CAAE: 64797816.7.0000.5542). As shown in Table 1, there is a variability not only in gender, but also on subjects' weight and height, which provides evidence for a further generalization of the proposed study.

**Table 1.** Characteristics of the participants of this research.

Two IMU sensors, one POF curvature sensor and two RGB-D cameras were used to estimate elbow joint angles. The IMU reference sensor was placed on the superior third of the right upper arm, and the second IMU was attached dorso-distally on the right forearm, as shown in Figure 6e. In a standing neutral posture, both sensors were positioned with *x*-axis pointing cranially, *z*-axis laterally and *y*-axis an orthogonal axis to *x* and *z* axes. These positions have been suggested by different authors [9,35,37]. Moreover, the POF curvature sensor was carefully aligned with the elbow joint in such a way that the sensitive zone of the optical fiber is located on the axis of rotation (flexion-extension axis).

In this test, the 11 participants (see Table 1) perform, a comfortable self-velocity, flexion-extension movements on three planes: sagittal, transverse, and frontal. Each participant was standing at the center of the room, observing the middle point between the two RGB-D cameras, see Figure 6d. All trials start with a synchronization movement, which consists of keeping the elbow in maximum extension on standing posture, then performing an elbow flexion of 90◦ and returning to the extended elbow position, where each transition last 5 s. Then, the subject was asked to perform three repetitions of flexion-extension on a specific plane. In the sagittal plane, the shoulder is in a neutral position and the participant performs elbow flexion-extension to get the maximum angle as possible (see in Figure 6a). In the transverse plane, the shoulder is in abduction (at max 90◦) and kept in that position for 5 s before the elbow flexion-extension

movements (Figure 6b). In the frontal plane, the shoulder is in abduction (at max 90◦) and external rotation, so the palm of the hand is facing forward as shown in Figure 6c. These described steps can be summarized in Figure 7.

**Figure 6.** Sensors' placement on the human upper limb, and movements performed during the experimental protocol. (**a**) User movement representation in sagittal plane, (**b**) transverse plane, and (**c**) frontal plane. (**d**) Top view of RGB-D system arrangement. (**e**) User using the IMU system and POF sensor.

**Figure 7.** Summary of the protocol's phases.

#### **3. Results and Discussion**

The comparison variables of the three systems were: (i) the correlation coefficient and (ii) the root mean squared error (RMSE) between the RGB-D cameras, IMUs, and POF. After the comparison among the sensors, a novel approach for angle correction on the markerless camera-based system was proposed and validated for the correction of angular errors on the sagittal plane. The proposed technique is based on anthropometric measurements of each subject, where the errors of the camera-based system compared with the POF curvature sensor and IMU system are corrected through a correlation between the measured errors and the length of the arm of each subject (*d*1, *d*2 and *d*3 of Equation (1)). In this way, an equation for error correction considering the length *d*3 of each subject was obtained.

#### *3.1. Sensors Characterization*

#### 3.1.1. IMUs and RGB-D Cameras

Figure 8 presents the results for RGB-D camera system and IMUs for the characterization phase using a goniometer as a reference for upper and lower bounds. The performed movement was divided into three cycles, and each cycle presents correlation between cameras and IMUs higher than 0.98.

**Figure 8.** Camera-based system and IMU angles of the first test.

Table 2 shows the maximum and minimum angles for cameras system and IMUs, compared to upper and lower bounds, respectively. The average error for the cameras system was 4.9◦, with a maximum error of 9◦, when compared with the goniometer for two values (90◦ e 20◦), which is lower than the mean error presented in Tannous et al. [38] (14, 6◦). However, in Tannous et al. [38] only one camera was used, carrying a higher self-occlusion leading to errors on the angle assessment. Since our system consists of two cameras, the self-occlusion decreases, consequently, reducing the errors. Compared to the goniometer, the IMUs average error was 3.7◦, which is lower than the camera-based system.

**Table 2.** Maximum and minimum angles of camera-based system and IMU of each cycle for the first test.

## 3.1.2. POF

Figure 9 presents the normalized POF power attenuation as a function of the angle on the experimental setup shown in Figure 6, where a high correlation with the measured angle was obtained, since the determination coefficient (R2) between the POF power variation and the angle was 0.996.

**Figure 9.** Sensors' placement on the human upper limb and the movements performed.

Thereafter, the POF curvature sensor is positioned on the subject's elbow joint as previously described (see Section 2.5). The data of the elbow movement at each plane (sagittal, frontal and transverse) were recorded, which are presented in Figure 10 for subject M3. The initial movement of 90◦ in the sagittal plane was used to adjust the sensor response with respect to the differences aroused from the positioning on the subject elbow.

**Figure 10.** Sensors' placement on the human upper limb and the movements performed.

The sensor response has the pattern of the flexion/extension movements performed by the subject. Furthermore, the angles estimated by the POF curvature sensor are within the range of movement described by the literature for elbow (0–145◦) [39]. Thus, the results presented in Figure 10 in conjunction with the high correlation of 0.996 (obtained on the sensor characterization as a function of the angle measured by a potentiometer) shows the feasibility of the POF sensor on the angle assessment, which makes it a suitable option for the comparison with angles estimated by the camera-based system.

#### *3.2. Comparison among the Sensors*

After the first evaluation of the sensors and the POF curvature sensor characterization, the sensors used in this work were compared using the experimental protocol depicted in Section 2.5. In this case, the camera-based system was compared with both IMU and POF. The comparison was made with respect to the correlation coefficient and RMSE (as also performed in the previous sections). Figure 11 shows the results obtained for all sensors in different planes, i.e., sagittal, transverse, and frontal planes, for subject M1.

**Figure 11.** Comparison among camera-based system, POF curvature sensor and IMU in the sagittal, frontal, and transverse planes, for subject M1.

The results presented in Figure 11 show a good correlation between the errors of the POF curvature sensor and IMU, especially on sagittal and frontal planes. Although we used the same number of cycles to compare the sensors, the period of each movement is different, due to the fact that each subject was allowed to perform the movements at a comfortable self-velocity.

Furthermore, the range of movement at each plane is different, i.e., the movement at the sagittal plane occurs in a range of about 0–145◦, whereas the one at the transverse plane reaches angles lower than 130◦. Similarly, the angles at the frontal plane can be as high as 145◦ (as in the sagittal plane). From the tests, the mean deviation between POF curvature sensor and IMUs was about 6.5% on the tests in the sagittal plane. However, such deviation increased to about 10% on the transverse and frontal planes. The reason for this increase can be related to the POF positioning on the tests, since it is a critical factor on the angle assessment using such technology. In addition, it can be also related to the increase of the errors of the IMUs when the test was performed in planes different from the sagittal one, as reported in Vargas-Valencia et al. [33]. Regarding the camera-based system, the results at the sagittal plane show an overestimation of the angle, when compared to IMU and POF curvature sensor. In this case, the angles estimated by the markerless camera system had a maximum value of about 160◦, which is higher than the elbow range of motion [39]. In contrast, the camera-based system underestimates the angles at the frontal plane when compared with the other two systems for angle assessment.

Such as aforementioned, the errors on the markerless camera system for angle assessment are related to issues, such as frame errors, exploitation of multiple image streams and, especially, due to self-occlusions. To further evaluate the errors obtained by the camera-based system, Table 3 presents the correlation coefficient and RMSE between the markerless system and the IMUs for each of the 11 participants in all three planes tested, whereas Table 4 presents the correlation and RMSE between the markerless system and the POF curvature sensor.

**Table 3.** RMSE and correlation coefficient for Cameras and IMU.

**Table 4.** RMSE and correlation coefficient for Cameras and POF.

Tables 3 and 4 show a correlation coefficient higher than 0.9 in all analyzed cases, which indicates a high correlation between the responses of the sensors. In addition, the standard deviation of the correlation coefficient was below 0.01 in all the analyzed cases. Thus, it is possible to verify not only a high correlation between the data of the camera-based system compared to the wearable ones, but also that the results present a promising evidence of repeatability of such systems. The mean of correlation coefficients between the camera-based system and the IMUs were 0.990, 0.984 and 0.979 on the sagittal, transverse, and frontal planes, respectively. It is noteworthy that higher correlations were obtained between the camera-based system and the IMUs than the ones comparing the markerless system with the POF curvature sensor. The mean of the correlation coefficients considering the later comparison was 0.978, 0.964 and 0.975 for sagittal, transverse, and frontal planes, respectively.

Even though the proposed camera-based system presented high correlation with the wearable sensors in all scenarios, the errors of such system are generally high. Such as can be observed in Figure 11, there are deviations on the angle estimation of the camera-based system when compared with the wearable sensors, where, considering all the performed tests, these errors can be as high as 15◦ on the worst case. In addition, the mean errors are about 10◦ when compared with the wearable sensors. It is noteworthy that these errors are lower than the ones reported on the literature [40], which is mainly due to the use of two cameras to reduce the errors related to occlusions. However, errors of about 10◦ are still not sufficient when a reliable system for movement analysis is concerned. Nevertheless, the high correlations obtained in all tests for the comparison with the wearable sensors (see Tables 3 and 4) indicate that the proposed markerless camera-based system can be a feasible solution for angle estimation if a post-processing technique for the correction of the angular errors is applied, as also discussed in Schmitz et al. [40].

#### *3.3. Technique for Angle Correction in Markerless Camera-Based Systems*

The primary assumption for the proposed compensation technique for angle errors in markerless camera-based systems is that the errors mainly occur due to occlusions, or errors on computer vision algorithm for the tracking of the anatomical points used to calculate the parameters *d*1 and *d*2 in Figure 4. If these parameters are incorrectly estimated, errors on the angle assessment will occur. Thus, it is possible to assume that these angular errors have a correlation with the anthropometric measurements of each participant. To verify this assumption and develop the compensation technique for the markerless system, each participant performed 3 flexion/extension cycles only on the sagittal plane (see Section 2.5), and the angles estimated with the markerless camera-based system were compared with the ones measured by the POF curvature sensor. We used the POF curvature sensor for the development of the compensation technique, since it was already evaluated with respect to the potentiometer, presenting low errors in this characterization. However, we must emphasize that other sensor systems can be used as reference for the proposed compensation technique, including IMUs, marker-based camera systems and goniometers. The technique proposed here is based on the basic premise that the errors are mainly related to the detection of the parameters *d*1, *d*2 and *d*3 (due to self-occlusions, numerical errors on the computer vision algorithm, among other reasons). Therefore, the errors can be correlated (and then compensated) by considering the actual value of the anthropometric measurements (*d*1 and *d*2) used on the angle estimation, which can be measured on each subject or estimated using the subject's height [41].

For the first characterization of the technique, the flexion/extension cycles of five subjects (M1, M3, M5, F2 and F5) are analyzed, and a polynomial regression between the angles estimated by the camera-based system and the POF curvature sensor is performed for each of the five subjects, where each equation has the type shown in Equation (7):

$$
\hbar \arg\_{ref} = a \cdot \arg\_{cam} \cdot^3 + b \cdot \arg\_{cam} \cdot^2 + c \cdot \arg\_{cam} + d,\tag{7}
$$

where *angcam* is the angle estimated by the camera, *angref* is the angle measured by the POF curvature sensor. In addition, a, b, c and d are polynomial regression coefficients experimentally obtained through the regression between the angular responses of both sensor systems, i.e., markerless camera-based and POF sensor, using the least squares method. The coefficient d is the offset on the sensor response (in ◦). Therefore, if the sensors responses are normalized in the beginning of the test, the offset will be null. For this reason, the coefficient d is not employed on the analysis of correlation between the coefficients of the angular error correction and the anthropometric measurements of the subjects.

Figure 12 shows the regression between the angle measured by the camera-based system and the POF curvature sensor for the third flexion cycle (as an example) of subject F5. The results show a high correlation (0.998) between the responses using a third-order polynomial regression. Actually, such high correlation occurs for all the cycles of the five subjects analyzed, where the correlation coefficient was higher than 0.9 in all cases. Hence, the assumption of correlation between the errors of both sensor systems holds true (based on the analyses performed). Then, the next step is to correlate the polynomial coefficients (a, b and c) with the anthropometric measurements of each participant.

**Figure 12.** Polynomial regression between camera-based system and POF curvature sensor angular responses for the third flexion cycle of subject F5.

As discussed in Section 2.1, the parameters used on the angle estimation by the camera-based system are the anthropometric distances (*d*1 and *d*2), which are detected through computer vision algorithms. Thus, errors on the detection of such points will lead to errors on angle estimation, where such errors can be related to those anthropometric distances. However, these parameters (*d*1 and *d*2) are intrinsic of each subject and can be easily measured. In addition, it is possible to use the height of each subject (as showed in Table 1) in conjunction with anthropometric data for males and females to correlate the arm length with the subject's height. Figure 13 shows the correlation of the polynomial regression coefficients (a, b and c) with the subjects' arm lengths (D).

**Figure 13.** Polynomial regression between the subjects' arm length (parameter D) and the coefficients of angular error correction.

The results as well as the equations presented in Figure 13 indicate the feasibility of using the anthropometric measurements of each subject on equations for angular errors corrections in camera-based systems. The correlation coefficient is higher than 0.9 for all analyzed coefficients, indicating the possibility of using the proposed compensation technique for angle correction. Then, by substituting the equations shown in Figure 13 in Equation (7), it is possible to obtain a corrected angle as depicted in Figure 14 for three flexion/extension cycles for subject F1. In addition, the uncompensated response, i.e., the response of the camera-based system without applying the equations for angle correction, is also presented for comparison purposes. The RMSE for the compensated response is also presented in order to verify the accuracy enhancement provided by the proposed technique. Compared to the uncompensated responses, where the RMSE was 15.04◦, 9.25◦ and 10.23◦ for cycles 1, 2, and 3, respectively, the proposed angular error compensation was able to reduce the errors substantially in all three cycles. To further verify the performance of the proposed technique, the aforementioned compensation equations were applied for the responses in the sagittal plane for all subjects. The comparison between the RMSEs for the cases with and without the compensation technique is presented in Table 5 for each subject in all three flexion/extension cycles analyzed, where the mean and standard deviation of the three cycles are presented for each participant.

The results presented in Table 5 show the feasibility of the proposed technique, where the RMSE was reduced for all 11 subjects analyzed. The highest reduction occurred in Subject F1, in which the RMSE reduced from 11.52◦ to 3.52◦ after applying the correction equations. The mean of the RMSEs for the compensated responses is about 4.90◦, whereas the uncompensated one is 10.42◦, which means a two-fold reduction of the RMSE when the proposed compensation is applied. It is also worth to mention that the lowest RMSE reduction for the compensated case occurred in subject M3, where the RMSE reduced 2.11◦. However, one should note that the RMSE of the uncompensated response of this subject was already low

(6.90◦) when compared to the ones of the other subjects and even when compared to the errors presented in the literature for similar systems [40].

**Figure 14.** Polynomial regression between the subjects' arm length (parameter *d*3) and the coefficients of angular error correction.

**Table 5.** Comparison between RMSEs of the compensated and uncompensated responses for each subject.

The proposed technique for angular errors correction in markerless camera-based system is a feasible and straightforward option to enhance the angular accuracy in such systems. There is a calibration step in which the response of the camera-based system has to be compared with the one of a reference sensor system, e.g., wearable or marker-based camera systems. Then, the errors obtained on the markerless camera-based system are compared with the subject's anthropomorphic parameters (arm length in this case) in order to obtain an equation that relates the angle correction with the parameters of each subject. Therefore, an important caveat should be mentioned: the calibration routine must be performed with respect to a reliable reference, and the movements should be performed at one plane, i.e., sagittal, frontal, or transverse planes movements. In addition, the calibration has to be performed on the same range at which the angle analysis will be performed, i.e., if an angular interval of 0 to 160◦ will be analyzed, the

calibration has to be made at this same angular range (0 to 160◦). By following these steps, it is possible to obtain accurate single plane angle measurements with a markerless camera-based system. Therefore, the main limitation of this approach is the necessity of a calibration stage prior to the application of the proposed sensor system in the same range and planes of movement envisaged on the proposed application. However, it is worth noting that the proposed approach can be extended for movement analysis of different degrees of freedom by adjusting the calibration stage accordingly and correlating the errors with the anthropometric parameters of each subject.


*3.2.2 External factors*

In general, environmental factors would have a very limited impact on dormancy. However, the effects of nitrogen, temperature and light were reported in the experiments. Nitrogen fertilization and relatively high temperatures would shorten the dormancy time. Whereas the light effect would have no significant impact.

The agricultural calendar for potato seed production should take into account the calendar for potato consumption production, the dormancy of tuber seed and the fact that tuber seed is generally earlier harvested when the tuber is still small. Therefore, three examples of calendars are given considering the dormancy duration of Potato varieties in CA countries (**Figures 3–5**), and where very early varieties, early and late ones cannot be scheduled in the same ways.


*Article* **Deep Learning Triplet Ordinal Relation Preserving Binary Code for Remote Sensing Image Retrieval Task**

**Zhen Wang 1,2,\*, Nannan Wu 1, Xiaohan Yang 1, Bingqi Yan 1 and Pingping Liu 2,3**

**Abstract:** As satellite observation technology rapidly develops, the number of remote sensing (RS) images dramatically increases, and this leads RS image retrieval tasks to be more challenging in terms of speed and accuracy. Recently, an increasing number of researchers have turned their attention to this issue, as well as hashing algorithms, which map real-valued data onto a low-dimensional Hamming space and have been widely utilized to respond quickly to large-scale RS image search tasks. However, most existing hashing algorithms only emphasize preserving point-wise or pairwise similarity, which may lead to an inferior approximate nearest neighbor (ANN) search result. To fix this problem, we propose a novel triplet ordinal cross entropy hashing (TOCEH). In TOCEH, to enhance the ability of preserving the ranking orders in different spaces, we establish a tensor graph representing the Euclidean triplet ordinal relationship among RS images and minimize the cross entropy between the probability distribution of the established Euclidean similarity graph and that of the Hamming triplet ordinal relation with the given binary code. During the training process, to avoid the non-deterministic polynomial (NP) hard problem, we utilize a continuous function instead of the discrete encoding process. Furthermore, we design a quantization objective function based on the principle of preserving triplet ordinal relation to minimize the loss caused by the continuous relaxation procedure. The comparative RS image retrieval experiments are conducted on three publicly available datasets, including UC Merced Land Use Dataset (UCMD), SAT-4 and SAT-6. The experimental results show that the proposed TOCEH algorithm outperforms many existing hashing algorithms in RS image retrieval tasks.

**Keywords:** remote sensing image retrieval; hashing algorithm; binary code; triplet ordinal relation preserving; cross entropy

#### **1. Introduction**

With the rapid development of satellite observation technology, both the amount and the quality of remote sensing (RS) images have improved dramatically. An era of remote sensing image big data has arrived. An increasing number of researchers are focusing on the task of large-scale RS image retrieval, due to its broad applications, such as disaster prevention, soil erosion monitoring, disaster rescue scenario and short-term weather forecasting [1–5]. The content-based image retrieval (CBIR) [6,7] method extracts feature information representing RS image content and finds similar RS images by comparing the distance values among their feature information. However, the feature information in CBIR is always represented as high dimensional float point data and it is difficult to directly compute the similarity relationship based on the original high dimensional feature information. Fortunately, hashing methods [1–5,8,9] can map high dimensional float point data into compact binary codes and return the approximate nearest neighbors according

**Citation:** Wang, Z.; Wu, N.; Yang, X.; Yan, B.; Liu, P. Deep Learning Triplet Ordinal Relation Preserving Binary Code for Remote Sensing Image Retrieval Task. *Remote Sens.* **2021**, *13*, 4786. https://doi.org/10.3390/ rs13234786

Academic Editors: Jukka Heikkonen, Fahimeh Farahnakian and Pouya Jafarzadeh

Received: 26 September 2021 Accepted: 23 November 2021 Published: 26 November 2021

**Publisher's Note:** MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Copyright:** © 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https:// creativecommons.org/licenses/by/ 4.0/).

to Hamming distance; this measure effectively improves the retrieval speed. In summary, the content-based image retrieval method assisted by hashing algorithms enables the efficient and effective retrieval of target remote sensing images from a large-scale dataset.

In recent years, many hashing algorithms [10–14] have been proposed to achieve the approximate nearest neighbor (ANN) search task, due to its advantage of computation and storage. According to the learning framework, the existing hashing algorithms can be roughly divided into two types: the shallow model [12–14] and the deep model [10,11,15,16]. Conventional shallow hashing algorithms, such as locality sensitive hashing (LSH) [14], spectral hashing (SH) [17], iterative quantization hashing (ITQ) [13] and k-means hashing (KMH) [12], have been applied to various approximate nearest neighbor search tasks, including image retrieval. Locality sensitive hashing [14] is a kind of data-independent method, which learns hashing functions without a training process. LSH [14] randomly generates linear hashing functions and encodes data into binary codes according to their projection signs. Spectral hashing (SH) [17] utilizes a spectral graph to represent the similarity relationship among data points. The binary codes in SH are generated by partitioning a spectral graph. Iterative quantization hashing [13] considers the vertexes of a hyper cubic as encoding centers. ITQ [13] rotates the principal component analysis (PCA) projected data and maps the rotated data to the nearest encoding center. The encoding centers in ITQ are fixed and they are not adaptive to the data distribution [12]. To fix this problem, k-means hashing [12] learns the encoding centers by simultaneously minimizing the quantization error and the similarity loss. KMH [12] encodes the data as the same binary code as the nearest center. For the image search task, the shallow model first learns the high dimensional features, such as scale-invariant feature transform (SIFT) [18] or a holistic representation of the spatial envelope (GIST) [19], then retrieves similar images by mapping these features into the compact Hamming space. In contrast, the deep learning model enables end-to-end representation learning and hash coding [10,11,20–22]. In particular, the deep learning to hash, such as deep Cauchy hashing (DCH) [11] and twinbottleneck hashing (TBH) [10], proves crucial to jointly learn, thereby similarly preserving the representations and control quantization error of converting continuous representations to binary codes. Deep Cauchy hashing [11] defines a pair-wise similarity preserving restriction based on Cauchy distribution and it heavily penalizes the similar image pairs with large Hamming distance. Twin-bottleneck hashing [10] proposes a code-driven graph to represent the similarity relationship among data points and aims to minimize the loss between the original data and decoded data. These deep learning to hash methods have shown state-of-the-art results for many datasets.

Recently, many hashing algorithms have been applied to the large-scale RS image search task [1–5]. Partial randomness hashing [23] maps RS images into a low dimensional Hamming space by both the random and well-trained projection functions. Demir et al. [24] proposed two kernel-based methods to learn hashing functions in the kernel space. Liu et al. [25] fully utilized the supervised deep learning framework and hashing learning to generate the binary codes of RS images. Li et al. [25] carried out a comprehensive study of DHNN systems and aimed to introduce the deep neural network into the large-scale RS image search task. Fan et al. [26] proposed a distribution consistency loss (DCL) to capture the intra-class distribution and inter-class ranking. Both deep Cauchy hashing [11] and the distribution consistency loss functions [26] employ pairwise similarity [15] to describe the relationship among data. However, the similarity relationship among RS images is more complex. In this paper, we propose the triplet ordinal cross entropy hashing (TOCEH) to deal with the large-scale RS image search task. The flowchart of the proposed TOCEH is shown in Figure 1.

**Figure 1.** Flowchart of the proposed TOCEH algorithm. Firstly, to represent the image content, we use the Alexnet, including five convolutional (CONV) networks and two fully connected (FC) networks, to learn the continuous latent variable. Secondly, the triplet ordinal relation is computed by the tensor product of the similarity and dissimilarity graphs. Thirdly, two fully connected layers with the activation function of ReLU are utilized to generate the binary code. To guarantee the performance, we define the triplet ordinal cross entropy loss to minimize the inconsistency between the triplet ordinal relations in different spaces. Furthermore, we design the triplet ordinal quantization loss to reduce the loss caused by the relaxation mechanism.

As shown in Figure 1, the TOCEH algorithm consists of two parts: the triplet ordinal tensor graph generation part and the hash code learning part. In part 1, we first utilize the AlexNet [27] pre-trained on the ImageNet dataset [28] to extract the 4096-dimension image feature information of the target domain RS images. Then, we separately compute the similarity and dissimilarity graph among the high dimensional features. Finally, we establish the triplet ordinal tensor graph representing the ordinal relation among any triplet RS images. Part 2 utilizes two fully connected layers to generate binary codes. During the training process, we define two excellent objection functions, including the triplet ordinal cross entropy loss and the triplet ordinal quantization loss to guarantee the performance of the obtained binary codes and utilize the back-propagation mechanism to optimize the variables of the deep neural network. The main contributions of the proposed TOCEH are summarized as follows:

The rest of this paper is organized as follows. Section 2 introduces the proposed TOCEH algorithm. Section 2.1 shows the important notation. The hash learning problem is stated in Section 2.2. The tensor graph representing the triplet ordinal relation among RS images is introduced in Section 2.3. We provide the formulation of triplet ordinal cross entropy loss and triplet ordinal quantization loss in Sections 2.4 and 2.5, respectively. The extensive experimental evaluations are presented in Section 3. Finally, we set out a conclusion in Section 4.

#### **2. Triplet Ordinal Cross Entropy Hashing**

*2.1. Notation*

In this paper, we use the letters B and X to separately represent the data matrix in the Hamming and Euclidean spaces. The columns in the data matrix are denoted as the letters with subscript. The important notations are summarized in Table 1.

**Table 1.** The important notations used in this paper.

#### *2.2. Hashing Learning Problem*

The purpose of the hashing algorithm [3,10,11] is to learn the hashing function *H*(·), mapping the high dimensional float point data *x* into the compact Hamming space as defined in Equation (1). *B*(*x*) represents the compact binary code of *x*.

$$B(\mathbf{x}) = (\text{sign}(H(\mathbf{x}) - 0.5) + 1)/2 \tag{1}$$

With the assistance of the obtained hashing function *H*(·), we can encode RS image content as compact binary code and efficiently achieve RS image search task according to their Hamming distances [1–5,23–25]. Furthermore, to guarantee the quality of the RS image search result, we expect the triplet ordinal relation among RS images in the Hamming space to be consistent with that in the original space [29,30]. To illustrate this requirement, a simple example is provided below. Here, *xi*, *xj* and *xk* separately represent RS image content information. In the original space, the image pair (*xi*, *xj*) is more similar than the image pair (*xj*, *xk*). After mapping them into the Hamming space, the Hamming distance of the data pair (*xi*, *xj*) should be smaller than that of the data pair (*xj*, *xk*). This constraint is defined as in Equation (2).

$$\begin{array}{lcl} & \left||H(\mathbf{x}\_{i}) - H(\mathbf{x}\_{j})||\_{1} \leq & \left||H(\mathbf{x}\_{k}) - H(\mathbf{x}\_{j})||\_{1} \\ \text{s.t.} & \left||\mathbf{x}\_{i} - \mathbf{x}\_{j}||\_{2}^{2} \leq & \left||\mathbf{x}\_{k} - \mathbf{x}\_{j}||\_{2}^{2} \end{array} \right. \end{array} \tag{2}$$

The constraint in Equation (2) guarantees that the ranking order of the retrieval result in the Hamming space is consistent with that in the Euclidean space. Thus, the hashing algorithm, satisfying the triplet ordinal relation preserving constraint, can achieve RS image ANN search tasks [31–35].

#### *2.3. Triplet Ordinal Tensor Graph*

To learn the triplet ordinal relation preserving hashing functions, the first problem is how to efficiently compute the probability distribution of the triplet ordinal relation among the training set in the original space.

Generally, we select the triplet data (*xi*, *xj*, *xk*) from the training set to compute their ordinal relation, where the data pair (*xi*, *xj*) has a small Euclidean distance value and (*xj*, *xk*) is considered as the dissimilar data pair. However, this mechanism needs to randomly select triplet samples and compare the distance values among all data points. It has a high time complexity and costly memory. Furthermore, it is difficult to define the similar and dissimilar data pairs for the problem without supervised information.

In this paper, to solve the above problem, we employ a tensor ordinal graph *G* to represent the ordinal relation among the triplet images (*xi*, *xj*, *xk*). We establish the tensor ordinal graph *G* by tensor production and each entry in *G* is calculated as *G*(*ij*, *jk*) = *S*(*i*, *j*)·*DS*(*j*, *k*). *S*(*i*, *j*) is the similarity graph as defined in Equation (3). A larger value of S(*i*, *j*) means the data pair (*xi*, *xj*) is more similar. *DS*(*i*, *j*) is the dissimilarity graph and its value is calculated as *DS*(*i*, *j*) = 1/*S*(*i*, *j*).

$$\mathcal{S}(i,j) = \begin{cases} 0, & i = j \\ e^{-||\mathbf{x}\_i - \mathbf{x}\_j||\_2^2 / 2\sigma^2}, & \text{otherwise} \end{cases} \tag{3}$$

We further process *G* to obey the binary distribution as in Equation (4). *gijk* is the entry of *G*(*i*, *j*, *k*).

$$\begin{cases} \mathcal{g}\_{i\bar{j}k} = 1, & \mathcal{G}(i, j, k) > 1 \\ \mathcal{g}\_{i\bar{j}k} = 0, & \mathcal{G}(i, j, k) \le 1 \end{cases} \tag{4}$$

Given *N* training samples, the size of the similarity graph and dissimilarity graph is *N* × *N*. The tensor product of the two graphs is shown in Figure 2, and its size is *N* 2 × *N* 2. However, the proposed TOCEH only concerns the relative similarity relationship among the data pairs (*xi*, *xj*) and (*xj*, *xk*). The corresponding elements are marked blue. There are *N* rectangles and each rectangle contains *N* × *N* elements. We pick up these elements and restore them into a matrix with the size of *N* × *N* × *N*.

**Figure 2.** The marked elements are picked up to restore in a matrix with the size of *N* × *N* × *N*.

Finally, the ordinal relation among any triplet items can be represented by the triplet ordinal graph G, as defined in Equation (5).

$$\begin{cases} \mathcal{S}(i,j) > \mathcal{S}(k,j), & \mathcal{g}\_{ijk} = 1\\ \mathcal{S}(i,j) \le \mathcal{S}(k,j), & \mathcal{g}\_{ijk} = 0 \end{cases} \tag{5}$$

To illustrate the cases defined in Equation (5), a simple explanation is provided below. For the triplet item (*xi*, *xj*, *xk*), the value of the (*ij*, *kj*)-th entry is *G*(*ij*, *kj*) = *S*(*i*, *j*)·*DS*(*k*, *j*) = *S*(*i*, *j*)/*S*(*k*, *j*). If the triplet ordinal relation is *S*(*i*, *j*) > *S*(*k*, *j*), we have *G*(*ij*, *kj*) > 1 and *gijk* = 1; otherwise, we have *G*(*ij*, *kj*) ≤ 1 and *gijk* = 0. Thus, the value in *G* can correctly indicate the true ordinal relation among any triplet items.

As described above, we can establish a tensor ordinal graph *G* with size *N*3 to represent the triplet ordinal relation among *N* images. In practice, during the training procedure, we use *L* (*L N*) k-means centers to establish the tensor ordinal graph, which can reduce the training time complexity.

#### *2.4. Triplet Ordinal Cross Entropy Loss*

In this section, we define *Gˆ* as RS images' triplet ordinal relation in the Hamming space. As discussed in Section 2.2, an ideal hashing algorithm should minimize the inconsistency between *Gˆ* and *G*. In this paper, the above requirement is achieved by minimizing the cross entropy value, as defined in Equation (6).

$$\min \text{CEH}(\text{G}, \stackrel{\frown}{\text{G}}) = \min -P(\text{G}) \log P(\stackrel{\frown}{\text{G}}) \tag{6}$$

*P*(*G*) defined in Equation (7) computes the probability distribution of RS images' triplet ordinal relation in the Euclidean space.

$$\begin{cases} w\_{ijk} = \frac{T\_1}{T} & g\_{ijk} = 1\\ w\_{ijk} = \frac{T\_0}{T} & g\_{ijk} = 0 \end{cases} \tag{7}$$

The definitions of *T*1, *T*0 and *T* are shown in Equation (8). *T*1 is the number of samples with a value of 1 in the matrix *G* and *T*0 is the number of samples with a value of 0 in the matrix *G*. *T* is the total number of the elements in the matrix *G*.

$$\begin{aligned} T\_1 &= \sum\_{i,j,k=1}^N \mathcal{G}\_{i,j,k} \\ T\_0 &= \sum\_{i,j,k=1}^N (1 - \mathcal{g}\_{i,j,k}) \\ T &= \sum\_{i,j,k=1}^N \left| 2 \cdot \mathcal{g}\_{i,j,k} - 1 \right| \end{aligned} \tag{8}$$

*P*(*Gˆ*) is a conditional probability of the triplet ordinal relation with given binary codes. As the samples are independent from each other, we calculate *P*(*Gˆ*) by Equation (9).

$$P(\widehat{G}) = \Pi\_{i,j,k=1}^{N} P(\mathcal{g}\_{ijk} \left| B\_{i\prime}, B\_{j\prime}, B\_k \right\rangle \tag{9}$$

*P*(*gijk*|*Bi*, *Bj*, *Bk*) is the probability of the triplet images satisfying the ordinal relation *gijk*, and the samples' are assigned the binary codes (*Bi*, *Bj*, *Bk*). The definition is shown in Equation (10).

$$P(\mathcal{g}\_{i\bar{j}k}|B\_{i\prime}B\_{\bar{j}\prime}B\_{\bar{k}}) = \begin{cases} \Phi(d\_h(B\_{k\prime}B\_{\bar{j}}) - d\_h(B\_{i\prime}B\_{\bar{j}})), & \mathcal{g}\_{i\bar{j}k} = 1 \\ 1 - \Phi(d\_h(B\_{k\prime}B\_{\bar{j}}) - d\_h(B\_{\bar{i}\prime}B\_{\bar{j}})), & \mathcal{g}\_{i\bar{j}k} = 0 \end{cases} \tag{10}$$

We further rewrite the definition of *P*(*gijk*|*Bi*, *Bj*, *Bk*) as in Equation (11).

$$P(\mathcal{G}\_{\vec{i}\vec{j}k}|B\_{i\cdot}B\_{\vec{j}\cdot}B\_{\vec{k}}) = \phi(d\_h(B\_{k\cdot}B\_{\vec{j}}) - d\_h(B\_{\vec{i}\cdot}B\_{\vec{j}}))^{\mathcal{G}\vec{i}\vec{j}} \left(1 - \phi(d\_h(B\_{k\cdot}B\_{\vec{j}}) - d\_h(B\_{\vec{i}\cdot}B\_{\vec{j}})) \right)^{1 - \mathcal{G}\vec{i}\vec{k}} \tag{11}$$

*dh*(·,·) returns the Hamming distance and *φ*(·) computes the probability value. If *gijk* = 1, the probability value should be close to 1 as *dh*(*Bk*, *Bj*)-*dh*(*Bi*, *Bj*) gets larger and the probability value should be close to 0 as *dh*(*Bk*, *Bj*)-*dh*(*Bi*, *Bj*) gets smaller. The characteristic of the function (·) is shown in Figure 3.

**Figure 3.** The characteristic of the function (·).

In this paper, the sigmoid function is considered as the function (·) as in Equation (12).

$$\Phi(d\_h(B\_k, B\_j) - d\_h(B\_i, B\_j)) = \frac{1}{1 + e^{-a(d\_h(B\_k, B\_j) - d\_h(B\_i, B\_j))}}\tag{12}$$

By merging Equations (7), (9), (11) and (12) into Equation (6), we reach the final triplet ordinal relation preserving objective function, as shown in Equation (13).

*L* = −*wijk* log Π*N i*,*j*,*k*=1*P*(*gijk Bi*, *Bj*, *Bk*) = *N* ∑ *i*,*j*,*k*=1 −*wijk* log *P*(*sijk Bi*, *Bj*, *Bk*) = *N* ∑ *i*,*j*,*k*=1 −*wijk* log ( 1 1+*e* −*α*(*dh*(*Bk* ,*Bj*)−*dh*(*Bi*,*Bj*)) ) *gijk* (1 − 1 1+*e* −*α*(*dh* (*Bk* ,*Bj*)−*dh*(*Bi*,*Bj*)) ) 1−*gijk* ) = *N* ∑ *i*,*j*,*k*=1 *wijk*(*gijk* log(1 + *e*−*α*(*dh* (*Bk* ,*Bj*)−*dh* (*Bi*,*Bj*)))+(1 − *gijk*)log(1 + 1 *e* −*α*(*dh*(*Bk* ,*Bj*)−*dh*(*Bi*,*Bj*)) )) = *N* ∑ *i*,*j*,*k*=1 *wijk*(*gijk* log(*e*−*α*(*dh* (*Bk* ,*Bj*)−*dh* (*Bi*,*Bj*))) + log(1 + 1 *e* −*α*(*dh* (*Bk* ,*Bj*)−*dh*(*Bi*,*Bj*)) )) (13)

#### *2.5. Triplet Ordinal Quantization Loss*

Generally, the sign function is adopted to map the real-valued data output by the last layer of deep neural network into binary codes. However, it generates discrete values and makes the objective function non-deterministic polynomial (NP) hard for optimization [20,36]. To fix this problem, the continuous tanh(·) function is utilized instead of the sign(·) function in this paper. Furthermore, to minimize the quantization loss caused by the continuous relaxation procedure, we expect the output of the tanh(·) function to be close to ±1. Here, we utilize the triplet ordinal cross entropy to formulate the quantization loss. We define the binary code obtained by the tanh(·) function as *Bi tah*. *Bref* is the reference binary code. The ideal encoding result is **1**. Thus, we formulate the quantization loss *Q* as in Equation (14).

$$\begin{split} Q &= \sum\_{i=1}^{N} -\log P(1 \mid (||B\_{\text{tail}}^{i}||, 1\_{\prime} || B\_{ref} ||)) \\ &= \sum\_{i=1}^{N} -\log \phi(-d\_{h}(||B\_{\text{tail}}^{i}||, 1) + \delta) \\ &= \sum\_{i=1}^{N} \log(1 + e^{-\mathfrak{a}(-d\_{h}(||B\_{\text{tail}}^{i}||, 1) + \delta)}) \end{split} \tag{14}$$

In Equation (14), the triplet ordinal relation among (||*Bi tah* ||, **1** and ||*Bref*||) is defined as 1 and it indicates that the data pair (||*Bi tah* ||, **1**) is more similar than

the data pair (**1**, ||*Bref*||). Therefore, to minimize the quantization loss, the Hamming distance of the data pair (||*Btah*||, **1**) should be smaller than the Hamming distance *δ* = *dh*(||*Bref*||, **1**). During the training procedure, we tune the value of *δ* to balance the optimization complexity and the approximation performance. A small *δ* value let the encoding results be close to the output of sign function and the training process will become hard. In contrast, a large *δ* value creates low optimization complexity, but it leads to poor approximation results.

After applying the continuous relaxation mechanism, we compute the Hamming distance of one data pair by Equation (15). ⊗ computes the sum of bitwise production value. *f* 8(·) represents the output of the deep neural network's last layer.

$$d\_h(B\_i, B\_j) = \frac{1}{2}(M - \tanh(f\_8(\mathbf{x}\_i)) \otimes \tanh(f\_8(\mathbf{x}\_j)))\tag{15}$$

Finally, we utilize the back propagation mechanism to optimize the variables of the deep neural network by simultaneously minimizing the triplet ordinal relation cross entropy loss in Equation (13) and the quantization loss in Equation (14).

#### **3. Experimental Setting and Results**

In this section, we introduce the comparative experimental setting and evaluate the approximate nearest neighbor search performance of the proposed TOCEH and some state-of-the-art hashing methods.

#### *3.1. Datasets*

The comparative experiments are conducted on three large-scale RS image datasets, including UC Merced land use dataset (UCMD) [37], SAT-4 dataset [38] and SAT-6 dataset [38]. The details of these three RS image datasets are introduced below.

Some sample images of the above three datasets are shown in Figures 4–6, and the statistics are summarized in Table 2.

#### *3.2. Experimental Settings and Evaluation Matrix*

To verify the ANN search performance of the proposed TOCEH method, many state-ofthe-art hashing methods, including locality sensitive hashing (LSH) [14], spectral hashing (SH) [17], iterative quantization hashing method (ITQ) [13], k-means hashing (KMH) [12], partial randomness hashing (PRH) [23], deep variational binaries (DVB) [39], deep hashing (DH) [40], DeepBit [41], deep Cauchy hashing (DCH) [11] and twin-bottle neck hashing (TBH) [10] are utilized as the baseline methods. LSH [14], SH [17], ITQ [13] and KMH [12] belong to the shallow methods. During the ANN search experiments, we extract the content information from RS images by AlexNet and the features are represented as 4096-dimension float point data. Then, these shallow hashing methods map the 4096-dimension features

into the compact Hamming space and achieve the ANN search task according to the Hamming distance. DCH [11], TBH [10], DVB [39], DH [40], DeepBit [41] and the proposed TOCEH are deep learning hashing methods. They directly generate the RS image's binary feature using an end-to-end mechanism.

**Figure 4.** Sample images of the UCMD dataset.

**Figure 6.** Sample images of the SAT-6 dataset.

The training process and comparative experiments are conducted on a high-performance computer with GPU Tesla T4 16 GB, CPU Intel Xeon 6242R 3.10 GHz and 64 GB RAM.

To evaluate the ANN search performance, two widely used standards, mean average precision (mAP) and recall curves, are employed in this paper.

The recall curve represents the fraction of the positive samples that are successfully retrieved. The definition of recall is shown in Equation (16). #(·) returns the number of samples.

$$recall = \frac{\#(retrieved\ positive\ samples)}{\#(all\ positive\ samples)}\tag{16}$$

Mean average precision value expresses the return rate of positive samples as defined in Equation (17). |*total*| is the total number of retrieved samples. Ki returns the number of positive samples of the *i*-th query sample. *rank*(*j*) is the ranking number of the *j*-th positive sample in the retrieved results.

$$mAP = \frac{1}{|total|} \sum\_{i=1}^{|total|} \frac{1}{K\_i} \sum\_{j=1}^{K\_i} \frac{j}{rank(j)}\tag{17}$$

#### *3.3. Experimental Results*

#### 3.3.1. Qualitative Analysis

In this section, we show the qualitative image search results on the UCMD dataset [37]. The proposed TOCEH and the other seven state-of-the-art methods separately map the image content information into 64-, 128- and 256-bit binary code. The images with minimal Hamming distance to the query sample are returned as retrieval results and the false images are marked with red rectangles, as shown in Figures 7–9.

**Figure 7.** The RS image retrieval results on the UCMD dataset, and the length of the binary code is 64. The false images are marked with red rectangles.

**Figure 8.** The RS image retrieval results on the UCMD dataset, and the length of the binary code is 128. The false images are marked with red rectangles.

**Figure 9.** The RS image retrieval results on the UCMD dataset, and the length of the binary code is 256. The false images are marked with red rectangles.

From the RS image retrieval results, we intuitively know that TOCEH owns the best retrieval results. When encoding RS image content as a 64-bit binary code in Figure 6, TOCEH and TBH [10] return two false positive images. Correspondingly, the number of

false images retrieved by the other six methods is larger than two. Furthermore, the false RS images' ranking position in TOCEH is higher than that in TBH [10], which gives TOCEH a larger mAP value. In Figure 7, the length of the binary code is 128. One RS image is incorrectly returned by TOCEH, TBH [10], DCH [11] and PRH [23], and the false image has a relatively higher ranking position in TOCEH. As the number of binary bits increases to 256, only TOCEH and TBH [10] retrieve no false image, as shown in Figure 8.

#### 3.3.2. Quantitative Analysis

In this section, we adopt *recall* curves and *mAP* to quantitatively analyze the ANN search performance of the proposed TOCEH and the other seven state-of-the-art hashing methods. These hashing methods separately generate 64-, 128-, and 256-bit binary code to represent the image content. The *mAP* values are in Tables 3–5. The recall curves are shown in Figures 10–12.

**Table 3.** Comparison of *mAP* with different binary code lengths on UCMD.

**Table 4.** Comparison of *mAP* with different binary code lengths on SAT-4.

**Table 5.** Comparison of *mAP* with different binary code lengths on SAT-6.

**Figure 10.** The recall curves of all comparative methods on UCMD; the data are separately encoded as (**a**) 64-, (**b**) 128- and (**c**) 256-bit binary code.

**Figure 11.** The recall curves of all comparative methods on SAT-4 and the data are separately encoded as (**a**) 64-, (**b**) 128- and (**c**) 256-bit binary code.

**Figure 12.** The recall curves of all comparative methods on SAT-6 and the data are separately encoded as (**a**) 64-, (**b**) 128- and (**c**) 256-bit binary code.

From the quantitative results, we know TOCEH achieves the best ANN search performance. LSH [14], the data-independent hashing algorithm, randomly generates hashing projection functions without a training process. As a result, the ANN search performance of LSH cannot drastically improve as the number of binary bits increases [9]. In contrast, the proposed TOCEH and the other nine comparative hashing methods utilize a machine learning mechanism to obtain the hashing functions, which are adaptive to the training data distribution. Thus, these machine-learning-based hashing algorithms achieve a better ANN search performance than LSH. SH [17] establishes a spectral graph to measure the similarity relation among samples, and divides the samples into different cluster groups by spectral graph partition. Then, SH [17] assigns the same code to the samples in the same group. For a large-scale RS image dataset, the time complexity of establishing a spectral graph would be high. Both ITQ [13] and KMH [12] first learn encoding centers, then assign the samples as the same binary code as their nearest center. ITQ [13] considers the fixed vertexes of a hyper cubic as centers, but they are not well adapted to the training data distribution.  This measure effectively helps KMH improve the ANN search performance. To balance the training complexity and ANN search performance, PRH [23] employs the partial randomness and partial learning strategy to generate hashing functions. LSH [14], SH [17], ITQ [13], KMH [12] and PRH [23] belong to the shallow hashing algorithms, and their performances relate to the quality of the intermediate high dimensional features. To eliminate this effect, TOCEH, TBH [10], DVB [39], DH [40], DeepBit [41] and DCH [11] adopt a deep learning framework to learn the end-toend binary feature, which can further boost the ANN search performance. The classical DH [40] proposes three constraints at the top layer of the deep network: the quantization loss, balance bits and independent bits. However, the pair-wise similarity preserving or the triplet ordinal relation preserving is not considered in DH. This may lead a poor performance of DH. The same problem also exists in DeepBit [41]. However, DeepBit

augments the training data with different rotations and further updates the parameters of the network. This measure helps DeepBit to obtain a better ANN search performance than DH. For most deep hashing, it is hard to unveil the intrinsic structure of the whole sample space by simply regularizing the output codes within each single training batch. In contrast, the conditional auto-encoding variational Bayesian networks are introduced in DVB to exploit the feature space structure of the training data using the latent variables. DCH [11] pre-trains a similarity graph and expects that the probability distribution in the Hamming space should be consistent with that in the Euclidean space. TBH [10] abandons the process of the pre-computing similarity graph and embeds it in the deep neural network. TBH aims to preserve the similarity between the original data and the data decoded from the binary feature. Both TBH [10] and DCH [11] aim to preserve the pair-wise similarity, and it is difficult to capture the hyper structure among RS images. TOCEH establishes a tensor graph representing the triplet ordinal relation among RS images in both Hamming space and Euclidean space. During the training process, TOCEH expects that the triplet ordinal relation graphs have the same distribution in different spaces. Thus, it can enhance the ability of preserving the Euclidean ranking orders in the Hamming space. As discussed above, TOCEH can achieve the best RS image retrieval results.

#### 3.3.3. Ablation Experiments

To guarantee the ANN search performance of the obtained binary codes, the TOCEH algorithm proposes two key components: the triplet ordinal cross entropy loss and the triplet ordinal quantization loss. Here, we conduct the comparative experiments to analyze these two components. TOCEL only utilizes the triplet ordinal cross entropy loss as the objective function for deep learning binary code. The deep hashing TOQL only employs the triplet ordinal quantization loss as the objective function. TOCEH, TOCEL and TOQL separately map the data into 64- and 128-bit binary code. The ANN search results are shown in Figures 13–15.

**Figure 13.** The ablation experiments on UCMD. The data are separately encoded as (**a**) 64- and (**b**) 128-bit binary code.

**Figure 14.** The ablation experiments on SAT-4. The data are separately encoded as (**a**) 64- and (**b**) 128-bit binary code.

**Figure 15.** The ablation experiments on SAT-6. The data are separately encoded as (**a**) 64- and (**b**) 128-bit binary code.

From the comparative results, we know that both the triplet ordinal cross entropy loss and the triplet ordinal quantization loss play important roles in improving the performance of TOCEH. The triplet ordinal cross entropy loss minimizes the inconsistency between the probability distributions of the triplet ordinal relations in different spaces. For example, the data pair (*xi*, *xj*) is more similar than data pair (*xj*, *xk*) in the Euclidean space. Then, to minimize the triplet ordinal cross entropy loss, it should be a larger probability to assign *xi* and *xj* as similar binary codes. Without the triplet ordinal cross entropy loss, TOQL randomly generates the samples' binary codes. LSH algorithm also randomly generates the hashing functions. Thus, the ANN search performance of TOQL is almost the same as that of LSH. To fix the NP hard problem of the objective function, we apply the continuous relaxation mechanism to the binary encoding procedure. Furthermore, we define the triplet ordinal quantization loss to minimize the loss between the binary codes and the corresponding continuous variable. Without the triplet ordinal quantization loss, the difference between the optimized variables and the binary encoding results would become larger in TOCEL. Thus, TOCEL has a relatively inferior ANN search performance. As discussed above, both the triplet ordinal cross entropy loss and the triplet ordinal quantization loss are necessary for the TOCEH algorithm.

#### **4. Conclusions**

In this paper, to boost the RS image search performance in the Hamming space, we propose a novel deep hashing method called triplet ordinal cross entropy hashing (TOCEH) to learn an end-to-end binary feature of an RS image. Generally, most of the existing hashing methods place emphasis on preserving point-wise or pair-wise similarity.

In contrast, TOCEH establishes a tensor graph to capture the triplet ordinal relation among RS images and defines the triplet ordinal relation preserving problem as the formulation of minimizing the cross entropy value. Then, TOCEH achieves the aim of preserving triplet ordinal relation by minimizing the inconsistency between the probability distributions of the triplet ordinal relations in different spaces. During the training process, to avoid the NP hard problem, we apply continuous relaxation to the binary encoding process. Furthermore, we define a quantization function based on the triplet ordinal relation preserving restriction, which can reduce the loss caused by the continuous procedure. Finally, the extensive comparative experiments conducted on three large-scale RS image datasets, including UCMD, SAT-4 and SAT-6, show that the proposed TOCEH outperforms many state-of-theart hashing methods in RS image search tasks.

**Author Contributions:** Conceptualization, Z.W. and P.L.; methodology, Z.W. and N.W.; software, P.L. and X.Y.; validation, N.W., X.Y. and B.Y.; formal analysis, Z.W. and N.W.; investigation, P.L. and X.Y.; resources, B.Y.; data curation, B.Y.; writing—original draft preparation, Z.W.; writing—review and editing, P.L.; visualization, N.W. and X.Y.; supervision, Z.W. and P.L.; project administration, Z.W. and P.L.; funding acquisition, Z.W. All authors have read and agreed to the published version of the manuscript.

**Funding:** This research was funded by the National Natural Science Foundation of China, grant number 61841602, the Natural Science Foundation of Shandong Province of China, grant number ZR2018PF005, and the Fundamental Research Funds for the Central Universities, JLU, grant number 93K172021K12.

**Acknowledgments:** The authors express their gratitude to the institutions that supported this research: Shandong University of Technology (SDUT) and Jilin University (JLU).

**Conflicts of Interest:** The authors declare no conflict of interest.

#### **References**


Herbs and Spices - New Processing Technologies

*Edited by Rabia Shabir Ahmad*

Published in London, United Kingdom

### *Supporting open minds since 2005*

Herbs and Spices - New Processing Technologies http://dx.doi.org/10.5772/intechopen.95216 Edited by Rabia Shabir Ahmad

#### Contributors

Nikhat Farhana, Eva Ivanišová, Vikrant Kumar, Deepak Mishra, Mukesh Chandra Joshi, Priyanka Mishra, Megha Tanwar, Prashant Kaushik, Rubi Gupta, Adeyemi Ojutalayo Adeeyo, Mercy Adewumi Alabi, Tshianeo Mellda Ndou, John O. Odiyo, Rachel Makungo, Erinfolami Motunrayo Enitan, Hosana Dumisani Mkoyi, Daniso Beswa, Diego Prado Vásquez, Rasmus Munk, Nabila Rodríguez Valerón, Vinod Kumar Paswan, Chandra Shekhar Singh, Garima Kukreja, Durga Shankar Bunkar, Basant Kumar Bhinchhar, Minoo Divakaran, N.T. Fathima Rafieah, Vinod Kumar Joshi, Apurva Joshi, Santosh Kumar Kar, Sitabja Mukherjee, Mavra Javed, Waqas Ahmed, Abdul Momin Rizwan Ahmad, Rehan Mian, Abebe Ayele Haile, Hency Rose, Yamini S, Aman Rathaur, Rabia Shabir Ahmad, Muhammad Sajid Arshad, Huda Ateeq, Muhammad Abdul Rahim, Muhammad Imran, Muhammad Kamran Khan, Muhammad Haseeb Ahmad, Miroslava Kačániová, Tatsiana A. Savitskaya, Dmitry D. Grinshpan

#### © The Editor(s) and the Author(s) 2021

The rights of the editor(s) and the author(s) have been asserted in accordance with the Copyright, Designs and Patents Act 1988. All rights to the book as a whole are reserved by INTECHOPEN LIMITED. The book as a whole (compilation) cannot be reproduced, distributed or used for commercial or non-commercial purposes without INTECHOPEN LIMITED's written permission. Enquiries concerning the use of the book should be directed to INTECHOPEN LIMITED rights and permissions department (permissions@intechopen.com).

Violations are liable to prosecution under the governing Copyright Law.

Individual chapters of this publication are distributed under the terms of the Creative Commons Attribution 3.0 Unported License which permits commercial use, distribution and reproduction of the individual chapters, provided the original author(s) and source publication are appropriately acknowledged. If so indicated, certain images may not be included under the Creative Commons license. In such cases users will need to obtain permission from the license holder to reproduce the material. More details and guidelines concerning content reuse and adaptation can be found at http://www.intechopen.com/copyright-policy.html.

#### Notice

Statements and opinions expressed in the chapters are these of the individual contributors and not necessarily those of the editors or publisher. No responsibility is accepted for the accuracy of information contained in the published chapters. The publisher assumes no responsibility for any damage or injury to persons or property arising out of the use of any materials, instructions, methods or ideas contained in the book.

First published in London, United Kingdom, 2021 by IntechOpen IntechOpen is the global imprint of INTECHOPEN LIMITED, registered in England and Wales, registration number: 11086078, 5 Princes Gate Court, London, SW7 2QJ, United Kingdom Printed in Croatia

British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library

Additional hard and PDF copies can be obtained from<EMAIL_ADDRESS>
Herbs and Spices - New Processing Technologies Edited by Rabia Shabir Ahmad p. cm. Print ISBN 978-1-83969-608-4 Online ISBN 978-1-83969-609-1 eBook (PDF) ISBN 978-1-83969-610-7


**Nickolas G. Kavallieratos**

Nickolas G. Kavallieratos is an Associate Professor of Agricultural Entomology and Acarology at the Agricultural University of Athens (AUA). During 2002-2014, he was a researcher at Benaki Phytopathological Institute (BPI) serving in all scientific ranks and administration positions (Head of the Laboratory of Agricultural Entomology, Head of the Department of Entomology and Agricultural Zoology, and Deputy Director) at BPI. He has also worked at several universities and institutes in Greece, Europe and Africa. Since 2009, he has been a frequent visiting scientist at the United States Department of Agriculture (USDA), Center for Grain and Animal Health Research, in Manhattan KS. Since 2001, he has also been a frequent visiting scientist at the Faculty of Biology, University of Belgrade. He studies morphology, morphometry, systematics, zoogeography, phylogeny, genetics and behavior of the Aphidiinae (Braconidae) parasitoids of aphids, providing descriptions of new species in science, redescriptions of species and genera, revisions of genera and reviews around the world (Europe, Asia, Africa, and North and South America) since 1994. The overall outcome of his research deals with the generation of tools that enable entomologists to identify Aphidiinae parasitoids with ease in different types of habitats, host plants and host aphids. He has also been working on the biology, ecology, demography, mating behavior and management of stored-product pests (insects, mites) with the use of natural (diatomaceous earths and other inert dusts), biological (entomopathogenic fungi, entomopathogenic nematodes and bacteria), chemical insecticides, and novel chemical or natural compounds (essential oils, nanoemulsions and microemulsion) that exhibit insecticidal properties. Prof. Kavallieratos has published >300 papers in peer-reviewed journals.


*Review* **Assessment of Response to Immunotherapy in Patients with Hodgkin Lymphoma: Towards Quantifying Changes in Tumor Burden Using FDG-PET/CT**

**Francesca Tutino 1, Elisabetta Giovannini 1, Silvia Chiola 2, Giampiero Giovacchini 1 and Andrea Ciarmiello 1,\***

**\*** Correspondence<EMAIL_ADDRESS>
**Abstract:** Immune checkpoint inhibitors are currently the standard of care for many advanced solid tumors, and they have been recently approved for the treatment of relapsed/refractory Hodgkin lymphoma and primary mediastinal B cell lymphoma. Assessments of the response to immunotherapy may be complicated by the occurrence of the flare/pseudoprogression phenomenon, consisting of initial tumor enlargement and even the appearance of new lesions, followed by a response, which may initially be indistinguishable from true progression. There have been efforts to characterize and capture the new patterns of response observed during immunotherapy, namely, pseudoprogression and delayed response, and several immune-related response criteria have been proposed. Confirming progression on a subsequent scan and measuring the total tumor burden are both common in immune-related criteria. Due to the peculiarity of hematologic malignancies, lymphoma-specific immune-related criteria have been developed (LYRIC), and they have been evaluated in research studies in comparison to the Lugano Classification. In this review work, we illustrate the evolution of the response criteria in lymphomas from the first CT-based criteria to the development of the PETbased Lugano Classification, further refined to take into account the flare phenomenon encountered during immunotherapy. We also describe the additional contribution of PET-derived volumetric parameters to the interpretation of responses during immunotherapy.

**Keywords:** immunotherapy; Hodgkin lymphoma; FDG-PET; pseudoprogression; LYRIC; MTV