In the article [8] they gift courses for the consumer in taking with fewer demanding situations primarily based totally on exceptional functions. As a result, the photograph calls for tons much less pre-processing to localize the document. This technique became now no longer very consumer-pleasant for the customers because of the constraints and slowdown of digitization. Article [9] used this technique for localizing. Following pre-processing, in addition, algorithms are required to complete the localization task. These algorithms can be divided into categories: 1. Use of more hardware 2. Consider strategies to apply gadgets imaginative and prescient. 3. The utility of deep gaining knowledge of algorithms. A scanning utility is presented [10] that consists of real-time web page recognition, fine assessment, and automated detection of a web page cover [11] while scanning books. Additionally, a transportable tool for putting the smartphone all through scanning is presented. Another paper that used extra hardware introduces a scaleinvariant characteristic remodel into the paper detection machine [12]. The hardware of the paper detection machine includes a virtual sign processor and a complicated programmable good judgment tool. The equipment can obtain and process images. The software program of this machine makes use of the SIFT technique to discover the papers. Compared to the conventional technique, this set of rules offers higher with the detection process. In the paper [13] paper detection desires a sheet of paper with a few styles revealed on it. It takes a laptop imaginative and prescient era one step toward getting used withinside the field.

#### **2.2.2 Machine Vision Techniques**

The set of rules [14] operates with the aid of using finding ability line segments from horizontal test strains. Detected line segments are prolonged or merged with neighboring test line textual content line segments to provide larger textual content blocks, which might be finally filtered and refined. The paper [15] provides a spotting device in complicated history video frames. The proposed morphological method [16] is insensitive to noise, skew, and textual content orientation. So it's miles without artifacts as a result of each constant/most excellent international thresholding and constant-length block-primarily based nearby thresholding. [17] proposes a morphology-primarily based totally method for extracting key assessment traits as guidelines for attempting to find appropriate license plates. Preferred license plates. The assessment function is in lighting fixtures adjustments and invariant to numerous variations like scaling, translation, and skewing. The paper [18] applies aspect detection and makes use of a low threshold to filter nontextual content edges. Then, a nearby threshold is chosen to each hold low-assessment textual content and simplify the complicated history of the excessive assessment textual content. Following that, textual content-region enhancement operators are proposed to emphasize areas with excessive aspect power or density.

[19] describes a step-with the aid of using-step method for locating candidate areas from the entered photo the use of gradient data, figuring out the plate region of the various candidates, and enhancing the region's border with the aid of using including a plate template. In the textual content extracts from video frames paper [20] the nook factors of the chosen video frames are detected. After deleting a few remoted corners, it merges the closing corners to shape candidate textual content areas. Target frames [21] are decided on at constant time durations from photographs detected with the aid of using a scene-extrude detection approach. A color histogram is used to perform segmentation with the aid of using color clustering surrounding color peaks for every selection.

The approach [22] locates candidate areas without delay withinside the DCT compressed area by the use of the depth version data encoded withinside the DCT area. The paper [23] makes use of a clean history for pix to locate the areas of interest (ROI). [24] proposes a linear-time line phase detector with dependable findings, a constrained quantity of fake detections, and no parameter tweaking. On a big set of herbal pics, our approach is evaluated and as compared in opposition to present-day techniques. Using Geodesic Object Proposals [26], a technique for detecting ability files [25] is defined in a given photo. The entered pics had been downsampled to the useful resource with the extraction of structures/functions of interest, to lessen noise, and to enhance runtime pace and accuracy. The outcomes indicated that the use of Geodesic Object Proposals withinside the file item identity activity is promising. Also, [27] operators are associated with the max-tree and min-tree representations of files in pix. In paper [28] a simple-towrite set of rules is proposed to compute the tree of shapes; When statistics quantization is low, it works for nD pics and has a quasi-linear complexity.

The methodology [29] is primarily based totally on projection profiles blended with a linked aspect labeling process. Signal cross-correlation is likewise used to affirm the detected noisy textual content regions. Several awesome steps are used for this challenge [30] a pre-processing system the use of a low-by skip Wiener filter, a difficult estimation of foreground areas, a history floor computation with the aid of using interpolating adjoining history intensities, the edge cost with the aid of using combining the computed history floor with the authentic photo consisting of the pinnacle test of the photo, and sooner or later a post-processing step to enhance the nice of the textual content regions and keep line connectivity. Removing the skew impact on digitalized files [31] proposed that every horizontal textual content line intersects a predefined set of vertical strains at non-horizontal positions. Just with the aid of using the use of the pixels on such vertical strains, we create a correlation matrix and calculate the file's skew attitude with terrific precision. In the challenge of white beard files [32] evolved a strong function-primarily based totally method to routinely sew a couple of overlapping pix. The cautioned approach [33] is primarily based totally on the combinatorial production of ability quadrangle picks from hard and fast line segments, in addition to projective file reconstruction with a recognized focal length. For line detection, the Fast Hough Transform [34] is applied. With the set of rules, a 1D model of the brink detector is presented. Three localization algorithms are given in an article [35]. All algorithms employ function factors, and of them moreover have a take a observe near-horizontal and near-vertical strains at the photo. The cautioned method [36] is a distinctly specific file localization approach for spotting the file's 4 nook factors in herbal settings. The 4 corners are about expected withinside the first step the use of a deep neural network-primarily based Joint Corner Detector (JCD) with an interesting mechanism, which makes use of a selective interest map to the kind of discover the file location.

#### **2.2.3 Machine Learning Method**

The paper [37] suggests a CNN primarily based approach as it should be localized files in real-time and version localization hassle as a key factor detection hassle. The 4 corners of the files are collectively expected with the aid of using a Deep Convolutional Neural Network. In the paper [38] first, stumble on the sort of the file and classify the images, after which with the aid of using understanding the sort of the file a matched localization approach is finished at the file and allows data extraction. Furthermore, another method offered a new usage of U-Net for file localization in pictures taken through smartphones [39].

## **3 Methodology**

We version the hassle of record localization as key factor detection. The approach desires a ground truth as a mask of the record component and the non-record component. We exhibit the record with white (255) and non-record components with black (0).

#### **3.1 Dataset Preparation**

We use [1–5] datasets as train and validation datasets and make the photo length comparable to the usage of the max photo top and width amongst snapshots the usage of 0 paddings. And use [6] dataset because of the check dataset for the assessment and evaluating the proposed approach with the preceding strategies and cellular applications.

#### **3.2 Using Deep Neural Networks**

For the mission of report localization in pix taken via way of means of smartphones, we used deeplabv3 [40] technique and the fine-tuning technique to retrain a few ultimate layers in the DeepLab neural network. This community has benefited from deconvolution. In this dissertation, we've taken into consideration locating the location of the report withinside the pix as a semantic segmentation mission. For convolutional neural networks, they've proven incredible overall performance in semantic photograph segmentation.

We use the possibilities of deeplabv3 to lessen the operational complexity via way of means of the usage of numerous neural community fashions withinside the semantic segmentation component like MobileNet [41]. The deeplabv3 with the mobilenetv2 has 2.11M parameters in total. We use mobilenetv2 as a function extractor in a simplified version of deeplabv3 to allow on-tool semantic segmentation. The resultant version achieves equal overall performance to the usage of mobilenetv1 as a function extractor (Fig. 1).

There are three layers in the DeepLab network. First, we emphasize atrous convolution [42], or convolution with up-sampled filters, as a useful technique in dense prediction problems. Throughout Deep Convolutional Neural Networks, atrous convolution allows us to directly regulate the resolution at which feature responses are calculated. It also enables us to efficiently expand the field of view of filters to include more context without increasing the number of parameters or calculation time.

**Fig. 1.** Deeplabv3 using mobilenetv2 neural network architecture

Second, we present an atrous spatial pyramid pooling (ASPP) method for segmenting objects at different sizes with high accuracy. ASPP uses filters at different sample rates and functional fields-of-views to probe an input convolutional feature layer, collecting objects and picture context at different levels. Third, we combine approaches from DCNNs and probabilistic graphical models to enhance object boundary localization. In CNN's, the frequently used combination of max-pooling and downsampling produces invariance but at the cost of accuracy in localization, we solve this by integrating the answers at the final DCNN layer with a fully linked Conditional Random Field (CRF), which has been proven to increase localization accuracy both qualitatively and statistically.

Because of the problem of semantic segmentation, there are large and small instances that need to be segmented. If convolution kernels of equal size are used, the problem may arise that the receptive field is not large enough and the accuracy of segmentation of large objects may decrease. As a reaction to this problem, at the historic moment an atrous convolution was created, i.e. The size of the dilation rate was adjusted to modify the convolution kernel's receptive field. The impact of atrous convolution on a branch convolutional neural network, on the other hand, is not beneficial. If we continue to use smaller atrous convolutions to recover the information of small objects, a large redundancy will result. ASPP uses the dilation rate of different sizes to capture information on different scales in the network decoder. Each scale is an independent branch. It is merged at the end of the network and then a convolution layer is an output to predict the label. This approach successfully eliminates the gathering of unnecessary data on the encoder, allowing the encoder to focus just on the object correlation.

The training level needs a different ground truth from the paper [6] so we provide a masked ground truth (Fig. 2) that the document part and the non-document part Are differentiated with black and white colors. The document with white (255) and the nondocument part with black (0). After freezing the intended layers, the final network has been updated and implemented in Ubuntu Linux version 16.04 LTS implementation and programming. The STRIX-GTX1080-O8G graphics card and the Core i7 6900k processor with 32 GB of RAM are also used for training, power, and network testing.

**Fig. 2.** Image sample with masked ground truth

## **4 Experiments and Results**

#### **4.1 Evaluation Protocol**

The IoU method described in [43], has been used to evaluate. First, the perspective effect is deleted from the ground truth (G) and predicted (S) with the help of image size. We call new situations (G- ) and (S- ) respectively so that the IoU or Jaccard index is equal to:

$$\text{IoU} = (\text{area}(\mathbf{G}' \cap \mathbf{S}'))/(\text{area}(\mathbf{G}' \cup \mathbf{S}')) \tag{l}$$

The final result is the average of the IoU value for each image.

#### **4.2 Results**

The result is compared with all well-known methods, algorithms, and mobile applications that can solve the document localization task in images. The other methods' results are compared based on [6] (Fig. 3).

**Fig. 3.** The result of the proposed method on dataset [6]

In Table 1, the final results in different categories are presented and Fig. 4 shows the result in comparison with the previous methods. We run the model on a check the dataset and look at our outcomes to the previously released outcomes on the same dataset [6]. While this check is sufficient to evaluate the tool's accuracy on the competition dataset, it does have drawbacks. One, it cannot show to us how effective our tool conflates to unknown contents because of the reality several samples of contents were used for education. Second, its miles now no longer capable of providing records about how well our framework generalizes to unseen documents with similar content material fabric.

We cross-validate our technique through manner of way of deleting each content material fabric from the education setting and then checking on the deleted content material fabric to recovery the weaknesses and diploma generalization on unseen content material fabric. The quit result is in comparison with all well-known methods, algorithms, and mobile packages that might solve the document localization venture in snapshots. The special methods' outcomes regions compared, based definitely on [6] (Fig. 3). Our technique successfully generalizes to unseen documents, as outcomes are validated in Fig. 4.

This is not unreasonable for the purpose that the low choice of the entered picture graph prevents our model from relying on features of the document's layout or content material fabric. We moreover end from the outcomes that our technique generalizes well to unseen smooth contents. It is crucial to mention that the technique is designed to be effective on a low huge style of reasserts like a midrange smartphone without the usage of cloud or server-based recourses. The frames from the four corners are processed within the order within the implementation. It can implement snapshots all at once through a manner of way of taking walks a batch of four snapshots via the model. This must result in a substantial boom in inefficiency.

**Table 1.** The result of the proposed method on dataset [6]

Simple Medium Difficult Average

**Fig. 4.** The result of the proposed method compared with previous methods

#### **5 Conclusion**

In this paper, we provided a new application of DeepLabv3 the use of MobileNetv2 for report localization in pics taken with the aid of using smartphones. The very last result is the exceptional result of those tasks' methods. We used all dependable datasets on this task. And the primary dataset to evaluate is the newly gathered dataset with a diverse variety of report localization challenges. And finally, we offer a utility the use of the Kivy framework.

We moreover go through some sensible techniques that can be carried out with the usage of software program software belongings like Python, PyTorch, TensorFlow, and OpenCV. We used all reliable datasets in this task. And, the number one dataset to study is the newly accumulated dataset with a various variety of document localization challenges. Also, we gift a unique method for locating documents in pictures. The problem of localization is modeled as a problem of key element detecting. We show that this method should make assumptions nicely on new and unseen documents via the usage of a deep convolutional network.

#### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Performance Comparison of Deep Residual Networks-Based Super Resolution Algorithms Using Thermal Images: Case Study of Crowd Counting**

Syed Zeeshan Rizvi(B) , Muhammad Umar Farooq, and Rana Hammad Raza

Electronics and Power Engineering Department, Pakistan Navy Engineering College (PNEC), National University of Sciences and Technology (NUST), Karachi, Pakistan<EMAIL_ADDRESS><EMAIL_ADDRESS>
**Abstract.** Humans are able to perceive objects only in the visible spectrum range which limits the perception abilities in poor weather or low illumination conditions. The limitations are usually handled through technological advancements in thermographic imaging. However, thermal cameras have poor spatial resolutions compared to RGB cameras. Super-resolution (SR) techniques are commonly used to improve the overall quality of low-resolution images. There has been a major shift of research among the Computer Vision researchers towards SR techniques particularly aimed for thermal images. This paper analyzes the performance of three deep learning-based state-of-the-art SR algorithms namely Enhanced Deep Super Resolution (EDSR), Residual Channel Attention Network (RCAN) and Residual Dense Network (RDN) on thermal images. The algorithms were trained from scratch for different upscaling factors of ×2 and ×4. The dataset was generated from two different thermal imaging sequences of BU-TIV benchmark. The sequences contain both sparse and highly dense type of crowds with a far field camera view. The trained models were then used to super-resolve unseen test images. The quantitative analysis of the test images was performed using common image quality metrics such as PSNR, SSIM and LPIPS, while qualitative analysis was provided by evaluating effectiveness of the algorithms for crowd counting application. After only 54 and 51 epochs of RCAN and RDN respectively, both approaches were able to output average scores of 37.878, 0.986, 0.0098 and 30.175, 0.945, 0.0636 for PSNR, SSIM and LPIPS respectively. The EDSR algorithm took the least computation time during both training and testing because of its simple architecture. This research proves that a reasonable accuracy can be achieved with fewer training epochs when an application-specific dataset is carefully selected.

**Keywords:** Crowd counting · Super resolution · Residual networks · EDSR · RCAN · RDN · Thermal imagery · PSNR · SSIM · LPIPS

#### **1 Introduction**

A normal human perception is limited to visible light spectrum only which ranges from ~380 nm to ~700 nm wavelength on the electromagnetic spectrum [1]. The visible cameras are designed to utilize the same wavelength range and replicate human vision by capturing RGB wavelengths for color representation. However, like a human eye, these systems are also affected by poor weather (such as fog, smoke, haze or storms etc.) and low illumination conditions. This limits their utilization to applications with only daytime scenarios. Beyond the visible spectrum is the infrared region which cannot be seen by humans. With the advancements in infrared/thermal imaging technologies, humans have extended their range of vision. These technologies enable the vision in most challenging situations such as low-visibility due to extreme weather conditions or low illumination etc. [2]. This is made possible because these cameras are essentially the heat sensors which can capture heat signatures from different objects. However, the spatial resolutions of these imaging technologies are relatively lower as compared to the visible cameras. Higher resolution is particularly important because it enables the capturing of small details in any image. The spatial resolution can be increased by use of a high-end camera, but that makes it a costly proposition. Many researchers use Image Super-resolution (SR) technique to reconstruct high resolution images from low resolution input images. SR is used to predict and fill details in low resolution images such that the output gives an image of a higher resolution. The resolution of the input image is increased based on the scaling factor used for super resolving the image.

SR is widely used in many computer vision applications including surveillance and security, small-objects detection and tracking to medical imaging. Most of the research regarding Image SR is focused towards images captured using visible cameras. However, surveillance environments are now commonly monitored using infrared/thermal cameras. The Computer Vision researchers are increasingly showing research interests in use of thermal images for a variety of applications [3–5]. Similarly, the same research trend is being noticed in SR applications using thermal images [6–9]. Rivadeneira et al. proposed a Convolutional Neural Network (CNN) based approach to compare performance of Image SR using both thermal and visible images [10]. The experimentation proved that the network trained with thermal images performed better than the latter.

This research work focuses on the use of deep Residual Network (ResNet) to perform SR of thermal images. ResNets have contributed significantly towards solving various Image SR issues. As presented in [11], a simple modification in traditional Convolutional Neural Networks (CNN) allows for bigger networks to be trained with an increased accuracy. The paper provides performance evaluation of some of the most popular ResNet architectures on the SR of thermal images. V. Chudasama et al. [12] provided a detailed comparison using Thermal Image Super Resolution (TISR) challenge dataset. The paper presented results of different state-of-the-art algorithms trained on this challenging dataset. Detailed analysis of large image datasets using state-of-the-art algorithms is though computationally expensive as it requires higher number of training epochs.

This research work specifically focuses on the application of crowd counting based on super-resolved thermal images using Enhanced Deep Super Resolution networks (EDSR), Residual Channel Attention Networks (RCAN) and Residual Dense Networks (RDN) algorithms. The algorithms were trained from scratch using fixed camera views obtained from two different video sequences (suitable for crowd counting application) of BU-TIV benchmark dataset. The super-resolved image outputs generated by these algorithms were then used to count the number of persons using pretrained model weights. To obtain ground truth of the person count, the total number of persons were also predicted on the original ground truth image using these weights. The main contributions of this paper are highlighted as follows:

The rest of the paper is organized as followed. Section 2 provides a brief overview of the related research. Section 3 presents the working methodology of compared algorithms, and provides details regarding the implementation of this research work. Section 4 discusses the results obtained from the experimentation. Finally, Sect. 5 concludes the paper and provides a future research direction.

#### **2 Related Work**

SR of thermal images has gained much of the interest among researchers working in this area. One of the earliest approaches to SR of thermal images made use of the Huber Total Variation (HTV) approach which employed Huber norm with bilateral Total Variation (TV) regularization [13]. Chen et al. proposed the use of visible camera to guide SR of thermal images [14]. The approach was tested on their dataset and showed a reasonable performance while also avoiding traditional over-texture problem. Hans et al. proposed an SR algorithm for thermal images based on sparse representation [15]. Their results showed a good performance of image reconstruction without introducing major counterfeit artifacts. Cascarano et al. proposed an SR algorithm which can handle both single and multiple images [16]. The algorithm was tested on aerial and terrestrial thermal images and showed a good performance.

The very first deep learning implementation for Image SR was presented by Dong et al. [17] in 2014, with the introduction of SRCNN. In 2016, residual networks were introduced by He et al. for image recognition [11]. Architectures based on residual networks were then explored for single image SR. This helped make significant advancements in this area. In [18], SRResNet was introduced which was a 16 blocks deep residual network. Improvements were made in the SRResNet architecture by Lim et al. [19] with the introduction of EDSR. These networks became the backbone of major future research work in the domain of single image SR using residual networks. A recent approach proposed in [20] was built on residual blocks as base units. The architecture was used to generate super-resolved images in ×2, ×3 and ×4 scales and showed a good generalization capacity.

As the SR of thermal images is a relatively new research area, there is a need to explore this area in detail and build motivation for future research using specific applications. Therefore, this research gives an overview of application-specific SR algorithms implemented on thermal images.

#### **3 Working Methodology of the Proposed Approach**

#### **3.1 Selected Algorithms**

Three different deep learning-based SR algorithms were used in this study for comparative analysis. All algorithms are built on ResNet-based architectures. As discussed in Sect. 1, ResNets are a special case of CNN where small modifications in traditional CNN networks such as addition of skip connections enable the training of larger networks. Surprisingly, the resultant larger network does not cause performance degradation as in the previous CNN networks and generates even better accuracies, which makes it a perfect candidate for thermal images SR.

SR algorithms used in this research include EDSR, RCAN and RDN. The EDSR network [19] is inspired from SRResNet, the first ever ResNet used for Image SR. It removed batch normalization layer in SRResNet which improved results. The RCAN [21] uses Residual in Residual network which allows training of very deep CNNs for Image SR with major performance improvements. The RDN [22] introduced Residual Dense Blocks (RDB) which extract local features using dense convolutional layers.

Similarly, Focal Inverse Distance Transform (FIDT) maps were used for localization and counting of crowd [23]. These maps accurately localize the crowd without head overlaps even in highly dense environments.

#### **3.2 Dataset**

The thermal images used to conduct this research work were extracted from video sequences of BU-TIV dataset [24]. Only two video sequences i.e., Marathon-2 and Marathon-4 were suitable for crowd counting application. Both sequences provided a good view of cameras fixed at an elevated platform to address the problem of crowd counting. Figure 1 shows example frames from Marathon-2 and Marathon-4 videos. Both video sequences have a far field view which adds complexities in crowd estimation process. Videos are selected to include both sparse and highly dense crowd environments. The crowd has both static and dynamic motion features in multiple directions. The skewed camera angles and fewer number of pixels per head further make these video sequences challenging for crowd analysis. Both selected video sequences are perfect choices for crowd analysis and provide opportunities to explore performance of SR with given challenging attributes.

The data was recorded on FLIR SC8000 cameras. A total of 3555 frames were extracted from these video sequences and then reshaped into 512 × 256 resolution. These were used as ground truth images. To construct low-resolution images for ×2 and ×4 factors, ground truth images were down sampled two and four times into images with 256 × 128 and 128 × 64 resolutions respectively. All images from both sequences were randomly shuffled to improve generalizability of the model. Dataset was then split into train, validation, and test sets with ratios of 80:10:10 respectively.

#### **3.3 Evaluation Setup**

The training was done using NVIDIA's GeForce GTX 1080Ti. The deep learning framework used was PyTorch. The training parameters were kept same for all SR algorithms. Learning rate was fixed at 0.0001, ADAM was used as an optimizer function with β1, β2 and ε set at 0.9, 0.999 and 10–8 respectively. The networks were trained using L1 loss function. All networks were trained with 16 residual blocks and 64 feature maps to keep the comparison fair. Training was continued until the L1 loss reached a numerical value of 1.5 for ×2 upscaling, and 3.0 for ×4 upscaling. The threshold values were selected based on the observation that the loss graphs started to plateau around these values. The weights were saved as soon as L1 loss reached the threshold value. These weights were then used to super-resolve the 355 test images by factors of ×2 and ×4. The average Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM) and Learned Perceptual Image Patch Similarity (LPIPS) values were calculated from test images. PSNR provides a ratio between maximum power of an image and power of corrupting noise that affects its quality. SSIM calculates how similar two images are by comparing luminance, contrast, and structure between two images. LPIPS evaluates distance between two image patches.

Crowd counting was done on super-resolved test images through FIDT maps. The code was run using PyTorch backend. Pre-trained weights from University of Central Florida - Qatar National Research Fund (UCF-QNRF) dataset [25] were used for this crowd counting estimation.

**Fig. 1.** Views from BU-TIV dataset (a) Marathon-2 (b) Marathon-4

#### **4 Results and Discussion**

The training results obtained for ×2 upscaling factor are shown in Fig. 2. The L1 loss for RCAN reached the threshold value of 1.5 in the least number of epochs i.e., 72. RDN took 94 epochs to reach the value, whereas the training for EDSR had to be early stopped at 199 epochs as the graph had plateaued. For ×4 upscaling factor, training was continued until the L1 loss value crossed the numerical value of 3.0. The training results are displayed in Fig. 3. In this case, the L1 loss for RDN was able to reach threshold value in only 51 epochs. For RCAN, it took 54 epochs. The graph for L1 loss of EDSR had started to plateau in this case too, which is why training was early stopped at 300 epochs.

Weights obtained from the reported epochs were used for the testing phase. 355 unseen images were used in the testing phase. The test results are displayed in Table 1. It can be clearly observed that reasonable scores are achieved by all algorithms even when the maximum training epochs were only 300. An SSIM score close to 1.0, LPIPS score close to 0.0 and a high PSNR value represents images with a high level of structural similarity and near to being highly identical. The visual comparison of results on test images for ×2 and ×4 upscaling factors are displayed in Fig. 4 and Fig. 5 respectively. The frames are evaluated using the PSNR, SSIM and LPIPS metrics. It can be observed that the algorithms have accurately predicted static features in the video frames, e.g., the parked cars and the road. The slight difference in scores observable in the results is because of the dynamic features, e.g., moving cars and pedestrians. However, all algorithms showed improvements in the results as compared to the bicubic interpolated versions of the same images. The table also shows runtime analysis of each algorithm for 355 test images. Generally, the performance of EDSR was not as good as RCAN and RDN, but it has a considerably shorter execution time per step because of the simpler architecture. It was observed that the time taken with ×2 upscaling factor was more than the time taken for ×4 upscaling factor using RCAN and RDN. This is because the input images for×2 upscaling factor are of a greater resolution than that of×4 upscaling factor, as previously discussed in Sect. 3. It was also observed that application-specific datasets with fixed camera views are computationally efficient and generate robust detections with considerably few epochs.

Crowd counting using FIDT maps was done on all super-resolved images obtained in the testing phase by each method. To establish ground truth, the pretrained weights were used for crowd counting on the ground truth images. The Mean Absolute Error (MAE) and Root Mean Square Error (RMSE) scores obtained by performing counting through bicubic up-sampling and selected SR methods are presented in Table 2. RCAN performed better than all other selected SR algorithms using both ×2 and ×4 upscaling factors. A visual comparison of crowd count between ground truth and the obtained results is also provided in Fig. 6 with both ×2 and ×4 upscaling factors.

**Fig. 2.** L1 loss and PSNR scores obtained during training of SR algorithms for ×2 factor

**Fig. 3.** L1 loss and PSNR scores obtained during training of SR algorithms for ×4 factor

**Table 1.** Quantitative comparison of selected SR algorithms on test data with×2 and×4 upscaling factors in terms of PSNR, SSIM and LPIPS metrics

**Table 2.** Performance evaluation of obtained crowd counting results with bicubic up-sampling and selected SR algorithms

(PSNR/SSIM/ LPIPS)

Bicubic (27.28/0.890/ 0.188)

EDSR (34.35/0.979/

RCAN (35.78/0.984/ 0.009)

RDN (36.19/0.985/ 0.011)

**Fig. 5.** Visual comparison of ×4 SR results for both static and dynamic features

**Fig. 6.** Comparison between ground truth and results obtained for all selected SR algorithms

## **5 Conclusion and Future Work**

This paper investigated the performance of state-of-the-art ResNet-based Image SR algorithms, namely EDSR, RCAN and RDN. The images were super-resolved ×2 and ×4 on video sequences of BU-TIV dataset. PSNR, SSIM and LPIPS scores were used as evaluation metrics to compare performance of each algorithm. As compared to the bicubic interpolated versions, all selected SR algorithms were able to generate good results due to their ResNet-based architectures which are proven to have good accuracies with deeper layers. With careful selection of a dataset with sufficient number of images for training, the models were able to perform good even with fewer epochs, the maximum of which were 300 epochs used by EDSR for ×4 up-scaling factor. The paper also provided a qualitative analysis by observing performance on crowd counting application in both sparse and highly dense crowd environments. RCAN outperformed other SR algorithms by achieving minimum MAE and RMSE values.

As a future work, a similar analysis can be performed with Image SR algorithms based on Generative Adversarial Networks (GANs). Furthermore, a completely new architecture can also be designed particularly tailored for crowd counting application using SR on thermal images. Similarly, different multi-image SR methods can also be explored for performance comparison with single image SR algorithms.

**Acknowledgement.** We acknowledge support from National Center of Big Data and Cloud Computing (NCBC) and Higher Education Commission (HEC) of Pakistan for conducting this research.

**Data Availability Statement.** All related data including dataset, trained model weights and highresolution results are placed on following Google Drive link: https://drive.google.com/drive/fol ders/1LNLVVNCRDRIP\_\_lN4DJSjxmws15BWcpw?usp=sharing (Last accessed on 15 Nov 21).

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Normalization and Bias in Time Series Data**

Aishwarya Asesh(B)

Adobe, Mountain View, USA<EMAIL_ADDRESS>
**Abstract.** Data normalization is an important preprocessing step in data mining and Machine Learning (ML) technique. Finding an acceptable approach to deal with time series normalization, on the other hand, is not an easy process. This is because most standard normalizing approaches rely on assumptions that aren't true for the vast majority of time series. The first is that all time series are stationary, which means that their statistical characteristics, such as mean and standard deviation, do not vary over time. The time series volatility is assumed to be uniform in the second assumption. These concerns are not addressed by any of the approaches currently accessible in the literature. This research provides theoretical and experimental evidence, that normalizing time series data, can prove to be of utmost value by trimming non necessary data points and achieving minimum information loss, by using the concept of Minimal Time Series Representation (MTSR).

**Keywords:** Time series · Normalization · Classification · Time series length · Minimal time series representation

#### **1 Introduction**

The University of California, Riverside (UCR) Time Series Classification Archive [1], has grown into a valuable resource for the time series data mining community, with over a thousand articles citing at least one data set from the repository. While the classification accuracy demonstrated by predictive models on UCR data is undeniable, it is critical to look into the impact of data normalization approaches on classification accuracy. Because data normalization procedures are known to have a substantial influence on prediction accuracy for many classifiers, a knowledge of the impact of UCR's approaches is required to validate the accuracy of classification models.

Due to the bias incorporated into time series classification approaches, created and evaluated on a single benchmark dataset, as discussed by Keogh and Kasetty [3], there is a clear need for broader testing on real-world data. However, it is important to validate if it's not just the data that causes methods to become over-trained, but also the normalization that goes into producing such datasets.

With the aid of raw unprocessed and non-normalized UCR data provided by Geoff Web, Anthony Bagnall and Eamonn Keogh, this research study focuses on normalization techniques and understanding the influence of normalization approaches on classifier (and regressor) accuracy.

#### **2 Normalization and Time Series Data**

Normalization methods are recognized to have a significant impact on classification accuracy in multivariate data sets. When data from two distributions with vastly different means and variances exist, normalization becomes critical in guaranteeing that each variable does not bias prediction. This may be less essential in univariate datasets [4,5]. Despite this, the multidimensional issue space becomes simpler to train in a variety of predictive models, including neural networks and support vector classifiers, and a number of mathematical functions rely on normalized data. The choice of activation functions in neural networks is greatly influenced by this fact, with sigmoid activations becoming essentially useless until input is in the 0–1 range. If the hyperplanes used in class separation can be fitted most precisely, support vector machines will require a standardised problem space. While this is a more complicated topic in and of itself, this aforementioned research concentrates on time series length standardisation.

While each data set in the UCR Time Series Data Archive has a drastically varied duration, it is crucial to identify the influence this has on categorization. The varying rates at which events occur throughout a number of occurrences is one reason why Dynamic Time Wrapping (DTW) distance measurements are so useful. When an event occurs in a specific length of time, it is possible that the same event will occur in a greater time frame in another occurrence, euclidean distance measurements will not match in these cases [6,7]. Although altering the time series length will not address this problem, it is critical in evaluating how much information is necessary in a specific time series to get best outcomes. The remainder of the raw time series is not necessary, if using a very short time series length is able too achieve high classification accuracy. This concept is comparable to early detection, which is a distinct field of time series classification. One can regulate which portion of the information and how much of it classifiers may utilize for prediction by altering the length of time series, both where they begin and where they stop [8,9]. While having more data provides for higher prediction accuracy, it also introduces noise, and the larger the data, the longer it takes to classify it. The information gain/loss, as well as the slow-down/speedup associated with it, may be understood through changes in time series length.

#### **3 Experiments**

The data from the UCR Time Series Classification Archive is not only standardized using z-score normalization, but it is also divided into training and testing subsets. The raw data required to create the UCR datasets includes CricketX, CricketY, CricketZ, GesturesX, GesturesY, GesturesZ, and Wafer. These datasets will be referred to as Cricket, Gestures, and Wafer. Two techniques are used to assess the accuracy of categorization on this data.

The first involves creating a distribution of classification accuracy for a particular data set using random train/test splits of the same size. The second technique is used to see whether there are any discrepancies between the raw data and the data in the UCR repository. To evaluate if there are any major discrepancies between the two datasets, each time series in the raw data is matched with its closest matching time series in the UCR repository data.

#### **3.1 Normalization**

**Scalar Normalization.** The data is normalized using the z-score method. The two most prevalent scale normalizing approaches, z-score normalization and minmax normalization, are used. When the data corresponds to a normal distribution, Z-score normalization is the most frequent and most representational of the original raw data. The process of Z-score normalization entails turning each data point into a positive or negative number that represents how many standard deviations the data point is from the mean.

Min-max normalization includes removing the minimum and dividing by the difference between the time series datapoint minimum and maximum values to transform data into values between 0 and 1. These normalization approaches are straightforward and widely used to eliminate bias in variables with larger values when compared to data with lower values.

**Time Series Length Normalization.** The data is organized so that each time series in a dataset is the same length. There is no need for stringent normalization techniques with regard to time series length, as in datasets like Cricket, the lengths of time series have a relatively little deviation from the mean. Despite this, when compared to the run durations of the closest neighbor technique, time series length minimization gives a significant computational speedup. The impact of both increasing and lowering the length of time series is an important aspect. Both approaches entail shrinking/expanding a *n-length* time series to a *m-length* time series.

For a time series *T* of length *n*, the *i*-th data point in *T* is represented as *Ti*. *T* is converted to a time series of length *m*, which is denoted as *S*. The new time series *S* where each data point in *S* is as follows:

$$S\_i = T\_j \quad j = \lfloor n \times \frac{i}{m} \rfloor$$

Information loss is observed when *m* is less than *n*, the impact of which will be detailed in the coming sections.

#### **3.2 Classification Techniques**

Using a variety of approaches, the UCR Time Series Classification Archive defines the lowest classification error achievable.

1 1-Nearest Neighbor classifier - 1-NN Euclidean - This is the error produced by utilizing one nearest neighbor algorithm with a euclidean distance metric.

The correctness of normalized data can be determined by comparing these three approaches. Because of their strength in time series classification [2], the universality and simplicity of the algorithms, these three approaches are chosen. While there are alternative approaches that may yield more accurate classifications, using NN classifiers allows for a better understanding of the homogeneity and relatedness of the normalized data within a particular class.

#### **4 Results and Discussion**

While the datasets are extremely diverse in origin, there are obvious parallels in terms of classification accuracy for both Euclidean and DTW distance metrics (Fig. 1).

**Fig. 1.** The distribution of time series length for each of the three datasets.

As the time series length grows, the classification accuracy approaches a maximum, as seen in Fig. 2 and Fig. 3. However, the maximum classification accuracy is obtained at a relatively short time series length and remains almost constant as the duration grows.

The concept of a Minimal Time Series Representation (MTSR), which minimizes data information loss (along with maximizing classification accuracy) is established. It can be observed that there is a minimum information loss in the instance of *Cricket* at a time series length close to the beginning values of the time series, 1200. While this reduces information loss, one can obtain the same classification accuracy with a time series length of 40 and a slightly greater classification accuracy distribution with a time series length of 90. The primary advantage of an MTSR is that it allows for quicker classification owing to the lower computing burden of conducting Euclidean distance measurements on a smaller time series.

**Fig. 2.** The distribution of classification accuracy for the *Cricket* dataset as a function of time series length using the Euclidean distance measure in NN prediction.

Similar results are observed for *Wafer* dataset, as illustrated in Fig. 3, where the classification accuracy stays steady as the time series length increases. One can detect a clear maximum classification accuracy for a length of 30 in this dataset. With a time series length of 10–20, a similar accuracy can be achieved as with a length of 300 (Fig. 4).

**Fig. 3.** The distribution of classification accuracy for the *Wafer* dataset as a function of time series length using the Euclidean distance measure in NN prediction.

**Fig. 4.** The distribution of classification accuracy for the *Gestures* dataset as a function of time series length using the Euclidean distance measure in NN prediction.

The results and findings utilizing a DTW distance measure differ considerably from those obtained using Euclidean distance measurements. There is a definite maximum classification accuracy in each of the three datasets. With the following optimum time series lengths and classification accuracies for each dataset:

There is a considerable absolute gain in classification accuracy in both the *Cricket* and *Gesture* datasets. While in case of *Wafer*, it appears to be merely a minor gain in classification accuracy, it is 0.77 standard deviations above the mean accuracy across all duration. This rise in accuracy over the mean is true for both *Cricket* and *Gesture*, with optimal time series length accuracy in *Cricket* X, Y, and Z being 1.66, 1.36, and 1.24 standard deviations above the mean, respectively. *Gesture* X, Y, and Z have ideal time series length accuracy of 1.12, 1.23, and 1.15 standard deviations above the mean, respectively. These are all considerable gains over average accuracy for various time series lengths, but the most interesting finding is that there is an ideal time series length for DTW accuracy. DTW has an unambiguous maximum with regard to time series length, unlike Euclidean measures, which have a minimal representation of the time series that nevertheless maximizes classification accuracy.

Because of the single maximum, there is considerable symmetry, with lengths on each side of the optimal length resulting in the same accuracy. In *Cricket*, for example, time series lengths of 30 and 550 produce extremely equal classification accuracy, but picking the length of 30 would be optimum for speed of calculation.

There is no simple answer to determining a generic minimal time series length, as there is with practically all sample size questions. It is determined by the number of model parameters to be evaluated as well as the quantity of data randomness. With the number of parameters to be estimated and the level of noise in the data, the sample size required grows (Figs. 5 and 6).

**Fig. 5.** The distribution of classification accuracy for the *Cricket* dataset as a function of time series length using the DTW distance measure in NN prediction.

**Fig. 6.** The distribution of classification accuracy for the *Cricket* dataset as a function of time series length using the DTW distance measure in NN prediction.

**Fig. 7.** The distribution of classification accuracy for the *Wafer* dataset as a function of time series length using the DTW distance measure in NN prediction.

## **5 Conclusions and Future Work**

While acknowledged to be less accurate than DTW in time series classification, Euclidean distance measurements have shown to be more stable than DTW. In comparison, DTW has a time series length that is optimal for highest classification accuracy.

The findings are limited to the datasets mentioned. While results about time series length and normalization approaches apply to various datasets, the nature of the data matters when it comes to classification accuracy. As a result, additional research is needed to determine the impact of time series length normalization and scalar normalizing on data from the UCR Time Series Classification Archive and other sources.

In addition, more research into multivariate time series is necessary. The information loss associated with the reduction in time series length has an impact on classification accuracy, as addressed in this paper (both positively and negatively). More study is needed to develop more complex models for determining the smallest time series representations with the least amount of information loss (Fig. 7).

**Acknowledgement.** All that I am, or ever hope to be, I owe to my angel mother.

## **References**

9. Chonbodeechalermroong, A., Ratanamahatana, C.A.: Robust scale-invariant normalization and similarity measurement for time series data. In: Sieminski, A., Kozierkiewicz, A., Nunez, M., Ha, Q.T. (eds.) Modern Approaches for Intelligent Information and Database Systems. SCI, vol. 769, pp. 149–160. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-76081-0 13

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Deep Neural Networks Approach to Microbial Colony Detection—A Comparative Analysis**

Sylwia Majchrowska1,2(B), Jaroslaw Pawlowski1,2, Natalia Czerep1,3, Aleksander G´orecki1,2, Jakub Kuci´nski1,3, and Tomasz Golan<sup>1</sup>

> <sup>1</sup> NeuroSYS, Rybacka 7, 53-656 Wroclaw, Poland<EMAIL_ADDRESS>

<sup>3</sup> University of Wroclaw, Fryderyka Joliot-Curie 15, 50-383 Wroclaw, Poland

**Abstract.** Counting microbial colonies is a fundamental task in microbiology and has many applications in numerous industry branches. Despite this, current studies towards automatic microbial counting using artificial intelligence are hardly comparable due to the lack of unified methodology and the availability of large datasets. The recently introduced AGAR dataset is the answer to the second need, but the research carried out is still not exhaustive. To tackle this problem, we compared the performance of three well-known deep learning approaches for object detection on the AGAR dataset, namely two-stage, one-stage, and transformer-based neural networks. The achieved results may serve as a benchmark for future experiments.

**Keywords:** Deep learning *·* Object detection *·* Microbial colony counting

## **1 Introduction**

The ability to automatically and accurately detect, localize, and classify bacterial and fungal colonies grown on solid agar is of wide interest in microbiology, biochemistry, food industry, or medicine. An accurate and fast procedure for determining the number and type of microbial colonies grown on a Petri dish is crucial for economic reasons - industrial testing often relies on proper determination of colony forming units (CFUs). Conventionally, the analysis of samples is performed by trained professionals, even though it is a time-consuming and error-prone process. To avoid these issues, an automated methodology based on artificial intelligence can be applied.

A common way of counting and classifying objects using deep learning (DL) is to first detect them and then count the found instances distinguishing between different classes. We compared the results of microbe colony counting using selected detectors belonging to different classes of neural network architectures, namely two-stage, one-stage, and transformer-based models.

Our experiments were conducted on the Annotated Germs for Automated Recognition (AGAR) dataset with *higher-resolution* subset [10], which consists of around 7k annotated Petri dish photos with five microbial classes. This paper focuses on setting benchmarks for the detection and counting tasks using stateof-the-art (SoTA) models.

#### **2 Object Detection**

Object detection was approached using two major types of DL-based architectures, namely two-stage and one-stage models. **Two-stage** detectors find classagnostic object proposals in the first stage, and in the second stage the proposed regions are assigned to the most likely class. They are characterised by high localization and classification accuracy. The largest group of two-stage models are Region Based Convolutional Neural Networks (R-CNN) [6], whose main idea is based on extracting region proposals from the image. Over the years, the networks from this family have undergone many modifications. In the case of *Faster R-CNN* [15] architecture, a Region Proposal Network (RPN) was used instead of the Selective Search algorithm. This allows for significant reduction of the model's inference time. In order to reduce issues with over-fitting during training, *Cascade R-CNN* [2] was introduced as multi-stage object detector, which consists of multiple connected detectors that are trained with increased intersection over union (IoU) thresholds. A year later, authors of *Libra R-CNN* [11] focused on balancing the training process by IoU-balanced sampling, balanced feature pyramid, and balanced L1 loss. In the following years, researchers used Faster R-CNN while replacing its backbone (CNN used as a feature extractor) with newer architectures. The most recent concept is *Composite Backbone Network V2* [8] (CBNetV2), which groups multiple pretrained backbones of the same kind for more efficient training.

On the other hand, the **one-stage** architectures are designed to directly predict classes and bounding box locations. For this reason, one-stage detectors are faster, but usually have relatively worse performance. Single-stage detection was popularized in DL mainly by You Only Look Once models (YOLO v1 [12], 9000 [13], v3 [14], v4 [1]), primarily developed by Joseph Redmon. Recently, the authors of *YOLOv4* [1] have enhanced the performance of YOLOv3 architecture using methods such as data augmentation, self-adversarial training, and class label smoothing, all of which improve detection results without degrading the inference speed. Moreover, the authors of *EfficientDet* [16] introduce changes which contribute to an increase in both accuracy and time performance of object detection. The main proposed changes include using weighted Bidirectional Feature Pyramid Network (BiFPN), compound scaling, and replacing the backbone network with EfficientNet [17]. EfficientNet as a backbone connects with the idea of scalability from compound scaling, which allows the model to be scaled to different sizes and to create a family of object detectors for specific uses.

Additionally, the **transformers** have recently become a next generation of neural networks for all computer vision applications, including object detection [3, 5,19]. The interest in replacing CNN with transformers is mainly due to their efficient memory usage and excellent scalability to very large capacity networks and huge datasets. The parallelization of transformer processes is achieved by using an attention mechanism applied to images split into patches treated as tokens. The utilization of the transformer architecture to generate predictions of objects and their position in an image was first proposed in DEtection TRansformer (DETR) network [3]. The architecture uses a CNN backbone to learn feature maps, then feeds transformer layers. In comparison to DETR, *Deformable DETR* [19] network replaces self-attention in the encoder and cross-attention in the decoder with multi-scale deformable attention and cross-attention. Deformable attention modules only attend to a small set of key sampling points around a reference point which highly speeds up the training process. The recently introduced *Cross-Covariance Image Transformers* (XCiT) [5] concept is a new family of transformer models for image processing. The idea is to use a transformer-based neural network as a backbone for two-stage object detection networks. XCiT splits images into fixed size patches and reduces them into tokens with a greater number of features with the use of a few convolutional layers with Gaussian Error Linear Units (GELU) [7] in between. The idea behind the model is to replace self-attention with transposed attention (which is over feature maps instead of tokens).

## **3 AGAR Dataset**

The AGAR dataset [10] contains images of microbial colonies on Petri dishes taken in two different environments, which produced *higher resolution* and *lower resolution* images. The differences are between the lighting conditions and apparatuses. *Higher resolution* images, which were used in our studies, can be divided into *bright*, *dark* and *vague* subgroups. On the other hand, considering the number of colonies, samples can be defined as *empty*, *countable* and *uncountable*. The dataset includes five classes, namely *E.coli, C.albicans, P.aeruginosa, S.aureus, B.subtilis*, while annotations are stored in json format with information about the number and type of microbe, environment and coordinates of bounding boxes.

In this paper, we present the results of experiments performed using a subset of the AGAR dataset, which consists of 6990 images in total. In our case only *higher resolution* (mainly 4000 *×* 4000 px), *dark* and *bright*, without *vague*, samples with *countable* number of colonies were chosen. Firstly, images were split into train and validation subsets (the same for each experiment), and then divided into 512 *×* 512 px patches as described in [10]. At the end—in the test stage—whole images from the validation subset of the Petri dish were used (for a detailed description of the procedure see Supplementary materials from [10]).

#### **4 Benchmarking Methodology**

We compared the performance of the selected models using several metrics: architecture type and size, inference time, and detection and counting accuracy.

During time measurements, the inference was executed on GeForce GTX 1080 Ti GPU using the same patch with 6 ground truth instances. The models were first loaded into memory, then inferred 100 times sequentially (ignoring the first 20 times for warming up) to calculate averaged time and its standard deviation for each model separately.

As to detection results, the detector performance was evaluated twofold - by measuring the effectiveness of detection and counting. As an evaluation metric for colony detection, we rely on the mean Average Precision (mAP), to be precise mAP@.5:.95, averaged over all 5 classes. The efficiency of colony counting was measured based on Mean Absolute Error (MAE), and Symmetric Mean Absolute Percentage Error (sMAPE).

With the growing popularity of DL, many open source software libraries implementing SoTA object detection algorithms emerge. Results provided for Faster R-CNN and Cascade R-CNN were taken from [10] for comparison purposes. Similarly, in our experiments we relied on MMDetection [4] framework (Libra R-CNN, CBNetV2, Deformable DETR, XCiT), Alexey Bochkovskiy's Darknetbased implementation of YOLOv4 [1], and Ross Wightman's PyTorch [18] reimplementation of official EfficientDet's TensorFlow implementation. To perform model training, we used the default parameters as for COCO dataset in the above-mentioned implementations. In all experiments, we used transfer learning technique based on pretrained backbones (ResNet-50, Double ResNet-50, XciT-T12) on ImageNet or whole architecture (YOLOv4, EfficientDet-D2, Deformable DETR) on MS COCO dataset. In the case of YOLOv4, we changed the input size to 512 *×* 512 px in order to match the size of the generated patches. We used pretrained backbones in all experiments. Traditional two- and one-stage networks were trained with Stochastic Gradient Descent (SGD) optimizer, as opposed to Transformer based architectures, where AdamW [9] optimizer was used. The values of initial learning rate vary between 10−<sup>3</sup> and 10−<sup>5</sup> for each model. All networks were trained until loss values saturated for the validation subset. We also chose commonly used augmentation strategies of selected models, like flips, crops, and resizes of images.

#### **4.1 Results**

Mean averaged precisions presented in Table 1 are averaged over all microbe classes. Calculated value of mAP@.5:.95 varies between 0.491 and 0.529. The most efficient results in terms of accuracy and inference speed were achieved for YOLOv4 architecture. On the other hand, transformer-based architectures present slightly worse performance. Some interesting cases were presented in Fig. 1. The selected image presents the same microbial species (*P.aeruginosa*), which forms two different sizes of colonies due to agar inhomogeneity, making detection even more challenging. Labeled small contamination is not perceived by all models (transformer based and EfficientDet-D2), and some of them (YOLOv4, Deformable DETR) also have problems with precise localization of blurred colonies. Two-stage detectors have a tendency to produce some excessive predictions.

**Table 1.** Benchmarks for tested models on the *higher-resolution* subset of AGAR dataset. The model size is given in terms of number of parameters (in millions). In case of XCiT model number of backbone's parameters is given in brackets.

**Fig. 1.** Examples of images of microbial species and achieved predictions for a selected sample. Whole image of the Petri dish presents ground truth annotations, while the white dashed rectangle indicates the region chosen for visualisation of predicted results. Red rectangles mark *P.aeruginosa* species, the black one – *Contamination* (microorganism not intentionally cultured), green – *C.albicans*, navy – *S.aureus*.

The performance of the selected architectures for microbial counting is presented in both Table 2 and Fig. 2, while Table 3 shows all five microbial species separately. In general, all detectors perform better for microbes that form clearly visible, separate colonies. The biggest problem with locating individual colonies was observed for *P.aeruginosa*, where the tendency for aggregation and overlapping is the greatest. Overall, the best results were obtained for the YOLOv4 model, where the predicted count of microbial colonies is the closest to ground truth in range from 1 to 50 instances (see Fig. 2) – the most operable scope for industrial applications. The straight black lines represent a 10% error to highlight the acceptable error range indicated by microbiologists. The worst performance was observed for the EfficientDet-D2 model – where small instances of microbial colonies were omitted (not localized at all), which may be caused by resizing the patches to fit the input layer size. Very low contrast between the agar substrate and the colony (*bright* subset of AGAR dataset) is an additional problem here.

**Table 2.** Symmetric Mean Absolute Percentage Error (sMAPE) and Mean Absolute Error (MAE) obtained for different models.

**Fig. 2.** The performance of microbial colony counting for 6 different models.

**Table 3.** Symmetric Mean Absolute Percentage Error (sMAPE) obtained for different types of microbes.

## **5 Conclusions**

In the conducted studies, we analyzed eight SoTA deep architectures in terms of model type, size, average inference time, and the accuracy of detecting and counting microbial colonies from images of Petri dishes. A detailed comparison was performed on AGAR dataset [10].

The presented results do not differ much between the different types of architectures. It is worth noting that we chose rather smaller, typical backbones for the purposes of this comparison to create a baseline benchmark for different types of detectors. It appeared that the most accurate (mAP = 0*.*529) and the fastest model (17 ms) is one-stage YOLOv4 network, making this model an excellent choice for industrial applications. Two-stage architectures of different types and kinds achieved moderate performance, while transformer-based architectures gave the worst results. EfficientDet-D2 turned out to be the smallest model in terms of the number of parameters.

Our experiments yet again confirm the great ability of DL-based approaches to detect microbial colonies grown in Petri dishes from RGB images. The biggest challenge here is the need to collect large amounts of balanced data. To train detectors in a fully-supervised manner, data must be properly labelled. However, identification of abnormal colonies grown in a Petri dish can be difficult even for a trained specialist - overlooked microbes were usually small and located at the edge of a Petri dish. Usually 10% error is an acceptable range for manual labeling. It is worth to point out that AI-assisted models can make predictions even for overcrowded dishes, which is a big advantage of this type of technology. They are not prone to fatigue like a human, however, variable lighting conditions can make detection even more difficult, which can be observed in our case for EfficientDet-D2 prediction for unrepresented *bright* samples.

**Acknowledgements.** Project "Development of a new method for detection and identifying bacterial colonies using artificial neural networks and machine learning algorithms" is co-financed from European Union funds under the European Regional Development Funds as part of the Smart Growth Operational Program. Project implemented as part of the National Centre for Research and Development: Fast Track (grant no. POIR.01.01.01-00-0040/18).

## **References**

106 S. Majchrowska et al.

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Low Voltage Warning System for Stand-Alone Metering Station Using AI on the Edge**

Krzysztof Turchan(B) and Krzysztof Piotrowski

IHP - Leibniz-Institut f¨ur innovative Mikroelektronik, Im Technologiepark 25, 15236 Frankfurt (Oder), Germany<EMAIL_ADDRESS>https://www.ihp-microelectronics.com

**Abstract.** Artificial intelligence is used in many different aspects of life these days. It is also increasingly used to create intelligent embedded devices. Their main task is to demonstrate intelligent features using limited hardware resources. This paper introduces the possibility of using AI on the Edge as a system to warn of low voltage on the battery in a standalone metering station powered by solar panels. The paper presents theoretical knowledge about artificial intelligence on the Edge, time series and algorithms in time series forecasting. In addition, a practical approach is presented in the Approach section. It also describes the stand-alone measurement station that was used, what type of data was collected and used, and what type of the time series forecasting algorithm was used. The main advantage of the approach is the practical use of Holt's Linear Trend method in battery voltage forecasting. Work on the project is still in progress therefore the results presented in the paper are generated during simulation.

**Keywords:** Artificial intelligence on the edge *·* Time series forecasting *·* Intelligent battery warning system

## **1 Introduction**

#### **1.1 Artificial Intelligence**

It is not easy to clearly define what is artificial intelligence (AI). AI could be defined as a set of algorithms which operation is similar to intelligent human behaviour [1]. All current AI-based systems belong to the so-called Narrow intelligence [2]. Narrow intelligence is one of three defined types of AI (Narrow AI, General AI, Super AI). This type of AI is designed to perform a single task, which can not be automatically applied to other tasks in other applications. Narrow AI-based systems can be more or less complex, depending on the computing power of the device they run on. There are two approaches to implementing AI. The first approach concerns the implementation of increasingly complex artificial intelligence systems to achieve the so-called General intelligence. The second approach is to implement AI algorithms on embedded devices, which are highly specialized devices designed for one or very few specific purposes and are usually embedded or included as part of a larger system. This type of AI is called AI on the Edge [3].

**AI on the Edge** is mainly related to above mentioned embedded devices. The idea behind this solution is to use the AI algorithms directly on the embedded device, without having to send large amounts of data to a server [3]. This solution offers many possibilities such as creating smart sensors, intelligent cameras, etc., by using various artificial intelligence methods depending on the purpose.

**Forecasting** is one of the most popular artificial intelligence methods whose main task is to predict future values based on data collected in the past [4]. Nowadays, predictions are made in every field where historical data is available. This is due to the fact that it gives the possibility to estimate different values, which often brings different kinds of benefits. Examples of using artificial intelligence for data prediction purposes include:

All of these examples have in common the fact that predictions are based on historical time-stamped data. This data is the domain of time series.

## **1.2 Time Series Forecasting**

Time series forecasting is the process of analysing time series data using modelling and statistics to make predictions. The accuracy of the prediction depends mainly on the available data. However, an important role is played by the proper selection of the algorithm for this data. There are many algorithms for predicting time series and they are selected according to several criteria.

The most important thing is to collect data with the same interval. This allows finding trends, seasonality or cyclic behaviour. Data gaps can be filled by algorithms but this makes the final model not as accurate as it could be [5].

#### **1.3 Forecasting Algorithms**

It is not always possible that one forecasting algorithm will work equally well for different types of plots (historical data). Very often it is necessary to select an algorithm based on observations of the data. This section introduces several time series forecasting methods, along with their purpose [6].

**Naive Method** is a technique that assumes that the next expected point is equal to the previously observed point, due to the not very large variability of the data over time. This method is not dedicated to highly variable data sets

**Simple Average** is a technique that fits with data that changes over a small period of time, while the average over a longer period remains constant [6].

**Moving Average.** Using a simple moving average model, we predict the next value (or values) in a time series based on the average of a fixed finite number of previous values [6].

**Holt's Linear Trend Method.** Any dataset that follows a trend can use this method for forecasting. Thanks to the method it is possible to predict future value using a linear trend. [7]. Holt's method has three separate equations that work together to generate a final forecast. The first Eq. (1) is a smoothing equation. It adjusts the last smoothed value to the trend of the last period. The trend is updated over time using Eq. (2). It involves determining the difference between the last two smoothed values.

$$L\_t = \alpha Y\_t + (1 - \alpha)(L\_{t-1} + T\_{t-1}) \tag{1}$$

$$T\_t = \beta (L\_t - L\_{t-1}) + (1 - \beta) T\_{t-1} \tag{2}$$

$$Y\_{t+1} = L\_t + (h)T\_t \tag{3}$$

where:

Y*t*+1 is value at time t+1, α and β are smoothing parameters, h is forecast horizon.

## **2 Motivation**

Applying the techniques and methods presented in the previous section, the idea was to implement a low battery warning system for a stand-alone measurement station.

As is well known, such measurement stations are often powered by solar panels because they could be located in places far from conventional power sources. This solution could generate problems because the panels that are connected to the measurement station could not be able to keep up with charging the batteries. Of course, it is possible to install panels with a much higher power than expected, but this generates additional costs.

Instead of that a low battery warning system has been designed. The exact purpose of this system is to predict the voltage of the battery a few days ahead and to warn when the voltage level may fall below the required one. Thanks to this the user responsible for the maintenance of the station is able to react to a warning at the appropriate time.

The second problem of stand-alone measurement stations is the possibility of losing access to the network. In this case, data that could not be sent must be stored in the embedded device's memory, which is often very limited. For this reason, it is important to send as little data as possible. The most important data that must be sent is information about the continuous operation of the station and warnings about possible battery discharge. Using the forecasting algorithm on the microcontroller makes it unnecessary to send all the electrical measurement data.

#### **3 Approach**

This section provides an approach to solving the problem presented in the previous section. Furthermore, it provides information on what type of data is collected on the microcontroller and an algorithm that is responsible for forecasting. It also gives ways to optimize algorithms to save as much microcontroller processing power as possible.

#### **3.1 Measurement Station**

The warning system covered in this paper is part of a larger measurement station as shown in Fig. 1 that collects data from the environment. The whole station is powered by a battery that is charged by solar panels. The station power supply section uses a Three-Channel (3-Ch) DC meter that records voltage and current measurements data for three independent channels. 3-Ch meter is a special purpose board designed at Institute for High Performance IHP, based on MSP430F6779 microcontroller.

The first channel that is shown in Fig. 1 measures voltage and current generated on the solar panel. This allows inferring the daily insolation level. Then, the second channel is used to measure battery voltage and current. With this data, it is possible to observe the charging and discharging cycles of the battery. The last channel serves to measure the current consumption of the environment measurement system. In addition, there is a sensor for collecting the temperature data. The schematic shows also an arduino MKR1010 WiFi board that serves as an network gate for the data.

**Fig. 1.** Measurement station schematic

#### **3.2 Dataset and Algorithm**

This entire dataset can be used to train an advanced artificial intelligence model such as Artificial neural network (such a case will be an object of future research). However, the application must run on an embedded device with relatively low computing power and it is important to use as simple algorithm as possible. The Fig. 2 below shows an example of the data collected on the 3-Ch DC meter. As can be seen in the figure (see Fig. 2), the battery voltage series shows a trend. The Holt's method described previously is ideally suited to predict battery voltage values. This gives the possibility to generate warnings if the predicted value is below the threshold value.

The microcontroller's task is to predict the battery voltage value for the next day based on the data from the five preceding days. In order to do this, the data must be prepared in advance. Each measurement is taken every 10 min. The predicted values will be daily values so in order to standardize the data with those that will be predicted it needs to be averaged beforehand. The data is averaged every 12 h which gives 2 values per day. This data is then stored in an array with the size of 10 elements. When this array is full, the first forecast of two 12 h values is made using the Holt's Linear Trend method. The value from the second forecast value is then checked. If this value is below the battery threshold then a low battery warning is given. Each day, the two oldest elements are removed from the array to make space for the two new values. The data in the array is shifted and the two new average values, calculated from the current day are written to the array. This is followed by a recalculation of the two forecast values for the following day. Then the algorithm is repeated over and over again. This gives a daily voltage forecast for the next day.

**Fig. 2.** Measurements collected on 3-Ch meter

The Fig. 3 shows a simulated forecast based on data that shows a downward trend. The data reflects a situation where the battery is continuously discharged by the load connected to it. It also represents the fact that battery voltage can be predicted using Holt's Linear Trend Method. The first version of the application was implemented in java programming language and executed on a PC. This was to verify the correctness of the prediction before implementing the algorithm on an embedded device. The plot shows a situation where the predicted voltage is close to 10 V. This would result in a low battery voltage warning. The prediction was made using data from the measuring station that was equipped with an under-sized solar panel on a cloudy day.

**Fig. 3.** Low voltage warning simulation

#### **4 Conclusions**

The whole idea of designing a low battery warning system based on artificial intelligence was to optimize the number of solar panels used in a stand-alone measuring station. Calculating the power demand and selecting the number of solar panels is nothing special, but here comes the aspect of random events such as a few days of bad weather. A warning system provides the user with sufficient time to react and secures the operation of the measuring station. As the implementation process is still underway, there are no calculated mean square errors of the forecast yet. Checking the correctness of the solution is the next task that will be performed. Unfortunately, it takes time to perform the test under natural conditions. Work will be continued directly on the microcontroller, which will give more accurate data on the speed of the proposed algorithm, needed RAM and processing power. After this, the next step will be to implement an algorithm that will use all the data described in the dataset and algorithm section. This will give the possibility to determine the prediction quality of the described solution.

Artificial intelligence is not just about complex systems for object recognition, it's also about simple algorithms implemented on devices that serve simple everyday tasks like collecting data from sensors.

**Acknowledgments.** This work was supported by the European Regional Development Fund within the BB-PL INTERREG V A 2014-2020 Programme, "reducing barriers - using the common strengths", by project SmartRiver, grant number 85029892 and by project SpaceRegion, grant number 85038043 and by the German government under grant 01LC1903M (AMMOD project). The funding institutions had no role in the design of the study, the collection, analyses, or interpretation of data, the writing of the manuscript, or the decision to publish the results.

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Test Framework for Collaborative Robot Graphene-Based Electronic Skin**

Jan Klimaszewski1(B) , -Lukasz Gruszka1,2 , and Jakub Mo˙zaryn<sup>1</sup>

<sup>1</sup> Department of Mechatronics, Institute of Automatic Control and Robotics, Warsaw University of Technology, ul. Sw. A. Boboli 8, 02-525 Warsaw, Poland

<EMAIL_ADDRESS><sup>2</sup> Easy Robots, Sp. z o.o., ul. Gdy´nska 32/14, Radom, Poland

**Abstract.** Collaborative robots are one of the key pillars of Industry 4.0. Thanks to improved sensors, they can cooperate with people in a common workspace safely. Equipping the robot with an electronic skin allows increasing its safety level. The article presents the hardware and software framework of the newly developed graphene-based electronic skin for a collaborative robot. Functional laboratory tests confirm the effectiveness of the e-skin integration with the robot control system.

**Keywords:** Collaborative robotics *·* Electronic skin *·* Graphene nanoplatelets *·* Force-sensitive resistors *·* Robot operating system

#### **1 Introduction**

The last decades brought new human-robot collaboration (HRC) areas in the same working space with examples in the industry, education, agriculture, healthcare services, security, and space exploration [1]. The collaborative robots (so-called cobots) are an important part of the 4th industrial revolution, better described as Industry 4.0 [7]. Unlike the well-known industrial robots, cobots are simple in operation, lightweight, reliable with enhanced sensors and easy to deploy. They are designed to follow special safety measures, handling human collision detection and avoidance contrary to closed industrial robotic cells.

Most cobots adopt compliant mechanisms and lightweight designs to reduce the impact force once collisions occur. Other solutions are power and force limiting (PFL) methods based on temporal dynamic models, and current signals from embedded torque sensors [8]. Compared with the above methods, equipping cobots with the electronic skin (e-skin) is a complementary way to increase their safety levels [4,5].

The sense of touch in robotic devices has been developed for many years. Mostly as a part of a gripper or an end-effector mechanism. Adding touch allows bio-mimicry to robots, as it has a very important defence function in nature. In animals, when the harmful factor causing the pain is triggered, the body automatically moves to avoid contact with the pain-causing factor. This movement is unconditional (it occurs automatically and is not subject to will). Recently, with the sensor miniaturization and new tactile technologies development, there is a possibility for the full robot body tactile e-skin manufacturing, and usage [4].

The paper aims to describe the developed hardware and software framework, combined in a laboratory test stand, allowing the functional tests of the newly developed graphene-based e-skin solution with the cobot. During tests, the e-skin was separated physically from the robot.

The paper is organized as follows. In Sect. 2, the prototype consisting of cobot, e-skin and software integration is presented. In Sect. 3, the preliminary tests are described and the obtained results are given. Finally, Sect. 4 provides the summary and further investigation proposal.

#### **2 Developed Test Framework**

The hardware framework of the developed laboratory test stand consisted of the e-skin based on graphene nanoplatelets and an ES5 robot from EasyRobots company<sup>1</sup>. For tests, the e-skin module was placed outside the working space of the manipulator. It was connected to the robot via a desktop computer with an Ethernet interface. Data exchange between the computer and the e-skin was based on UART communication using the USB port. The software framework was developed using Robot Operating System (ROS<sup>2</sup>) [2,6].

#### **2.1 E-skin**

The e-skin was constructed of force-sensitive resistors (FSR) connected in a rectangular matrix registering a pressure map from the touch exerted on it.

The e-skin consisted of two layers, presented in Fig. 1 [4,5]. The first one is a conductive layer of comb electrodes (1b-1) printed on plastic foil and connected along columns and rows. The second one consisted of FSR sensors arranged in a rectangular pattern (1b-2) placed on a plastic foil. In the research, we have used FSR matrices with the size of a single sensor approx. 5 *×* 5 mm. In order to present the touch measurement, a matrix of FSR sensors with the dimensions of 16 *×* 32 cells was used.

The FSR matrix combined the individual FSR sensors into columns and rows. In order to estimate the location of the pressure applied to the FSR matrix, each of the columns and rows was separately, sequentially powered by a dedicated electronic controller supplying *Vcc* = 3*.*3 V to only one sensor. The output *U<sup>x</sup>* (see Fig. 1a) of the matrix was connected to the ground via a measuring resistor. Accordingly, a circuit consisting of only one column and one row could be closed by the selected FSR sensor pressed to a row and column comb electrodes.

<sup>1</sup> https://easyrobots.pl/.

<sup>2</sup> https://www.ros.org/.

Then it was possible to read the measuring signal *U<sup>x</sup>* from the selected sensor. It has been estimated that as a result of the pressure, the measured FSR resistance changed in the range of 300 Ω*−*3*.*5 kΩ. Each FSR was based on graphene nanoplatelets similar to that described in [3]. The FSR matrix was connected with comb electrodes to detect the place and the contact force exerted thereon.

**Fig. 1.** a) Measurement principle for FSR matrix of size 2 *×* 2 [4]; b) the e-skin [5]: 1 a conductive layer of comb electrodes printed on plastic foil, 2 - FSR sensors arranged in a rectangular pattern placed on a plastic foil.

#### **2.2 Collaborative Robot**

We have tested the e-skin prototype using an ES5 robot. It is an industrial 6-axis robot with a ROS-based communication interface, mainly used in the metalworking industry, for palletizing, packaging and welding (Fig. 2c).

#### **2.3 Software Integration**

In the research, we have used a data exchange protocol based on publishing and subscribing messages provided by ROS. As part of the software framework, the ROS node was developed, allowing the operation of the e-skin with the robot. The data package was cyclically sent (with a frequency 50 Hz) in the form of a two-dimensional matrix of numbers corresponding to the e-skin prototype surface fields, with values reflecting the pressure level. A separate ROS node associated with the robot control system was analyzing the received values seeking maximum values. If it exceeded a fixed threshold value, the node sent the information into a robot control system to slow down the robot movements.

**Fig. 2.** The e-skin software (a), robot in simulation mode (b), a real ES5 industrial robot (c).

## **3 Tests**

#### **3.1 Research Method**

We carried out initial research using a robot simulation mode and a prototype of the e-skin module. We observed the interaction of the developed e-skin prototype (Fig. 2a) with the robot system using visualization in RViz (ROS Visualization, see Fig. 2b).

The second stage of the research was the integration of the device with the real robot. We have tested the correctness of the algorithm for forcing the robot to switch to slow speed mode in the effect of human contact with the surface of the e-skin and finding the robot's reaction time to a change in the state of the signal transmitted from the e-skin module<sup>3</sup>.

In both research stages, the robot moved along an experimental trajectory consisting of sections of straight lines run through the tool centre point (TCP) with a maximum speed of 1 m/s. Then, in response to human contact with the e-skin prototype matrix, the robot was forced to slow down to 250 mm/s (TCP speed, following the guidelines in point 5.7.3 of the PN-EN ISO 10218-1 standard, defining the collaborative robot reduced speed).

#### **3.2 Research Results**

Figure 3 shows the result of the robot reaction time to the pressure from the touch exerted on the e-skin module. It indicates the moments of pressure detection, receiving this information by the robot, and achieving the reduced speed. The individual times are shown in the Table 1. It can be seen that the total time from the detection of the force applied to the e-skin until the robot reaches the reduced speed is 0.414 s.

<sup>3</sup> Movie from the experiment is published on the page: http://www.jakubmozaryn. esy.es/?page id=2418.

**Fig. 3.** Reaction times for pressure detection

**Table 1.** Time reactions for e-skin pressure detection.

#### **4 Summary**

The paper describes the hardware and software framework for the newly developed graphene-based e-skin connected with an ES5 robot manipulator. The eskin was connected to the robot main computer unit within the Robot Operating System. The functional tests on a laboratory stand showed that the robot could be forced to slow down to 250 mm/s for the TCP speed in response to human contact with the e-skin. The presented tests prove that the developed system can be used for further research of the developed e-skin technology, as it also allows for new ways of interacting with electrical devices of daily use and can improve safety in traffic communication when introduced as an additional body coating.

The developed test framework will be the basis for further integration of the robot control system with the e-skin, which will allow for safe human-machine cooperation. A more elaborate robot response based on human touch is possible but requires improvements in the presented system's performance. In further works, the analysis and optimization of measurement and computational delays should be considered. The robot's reaction time is currently too low for human to work with it comfortably. In terms of the robot's response to human touch, the reaction time should be shortened to millisecond values.

Another part of further research will also cover the integration of the e-skin module with the robot control system by covering the whole surface of the robot structure with the flexible e-skin shell. For this purpose, a flexible cover with the applied sensory system, the same as the prototype described in this work, will be printed on the elastic mat, and a unique wiring system will be developed.

Finally, in presented research, we do not model the current flowing through sensors. It was observed that if many e-skin sensors are pressed simultaneously, the largest current will flow through the selected sensor, and a smaller current will also flow through the other sensors. It can lead to the ambiguity of the readouts. However, tests demonstrating the operation of the e-skin module have shown that it is still possible to determine the location and estimate the value of the pressure applied to the selected FSR. Addressing this problem is planned as part of the further e-skin module development.

#### **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **Realization of a Real-Time Decision Support System to Reduce the Risk of Diseases Caused by Posture Disorders Among Computer Users**

Enes Gumuskaynak1, Faruk Toptas1, Recep Aslantas1, Fatih Balki1, and Salih Sarp2(B)

<sup>1</sup> Aurora Bilisim, Istanbul, Turkey {enes.gumuskaynak,faruk.toptas,recep.aslantas<EMAIL_ADDRESS><sup>2</sup> Virginia Commonwealth University, Richmond, USA<EMAIL_ADDRESS>
**Abstract.** Nowadays, diseases caused by posture disorders are becoming more common. This situation reduces the working efficiency of people, especially computer users. This study aims to provide prevention of diseases caused by posture disorders faced by computer users and realize an application software to reduce disease risks. With this realized application, computer users' movements are monitored through the camera, and the situations that may pose a risk of disease for the users are determined. Realized application software is a decision support system. This decision support system provides users suggestions to change their position according to their instant postures and supports them to work more efficiently. The user data is collected by processing the images taken from a camera using the developed computer vision algorithm. Two-dimensional (2D) human exposure estimation is performed with the obtained data. The situations that can decrease the working efficiency are specified with the data obtained from exposure estimation using the developed model. As a result of these findings, increasing the working efficiency is provided by informing the user in real-time about the situation that may decrease the working efficiency.

**Keywords:** Computer vision with deep learning · Computer vision with pose estimation · Decision support system

#### **1 Introduction**

At the point brought by technology, the computer has become one of the indispensable parts of our lives. The time spent in front of the computer increased with the change of both remote working and education [1]. Depending on this situation, diseases such as tendinitis, myofascial pain syndrome, neck hernia, pulmonary embolism, eye strain, and humpback caused by postural disorders continue to increase and decrease the quality of life and work efficiency [2]. Recent advances in Artificial Intelligence (AI) provide new approaches in many industries [3, 4]. In this study, pose estimation is made by processing the images taken from the computer's camera to prevent the mentioned diseases. The pose estimation was developed with reference to the OpenPose [5] study, which was developed using deep neural networks and shared as open-source in python language. This study uses a bottom-up approach. Although this approach works slower than the topdown approach, it has been preferred. After all, it gives more accurate results because it can capture spatial dependencies between different people that require global inference. Posture analysis is performed using values obtained from the pose estimation study by the authors in [6]. The decision support system works by sending notifications to the user which also appears in the notification center in the case of situations that pose a risk.

In order to increase the efficiency of the pose estimation process, it is sufficient to determine only the eyes, ears, nose, shoulders, and neck to be used in the study. For this reason, a new data set containing the necessary body parts from the MPII Human Pose [7] dataset is curated. This study's customized pose estimation network is obtained by finetuning the model in [5] with the dataset, using the transfer learning method. Realtime posture analysis is performed using the obtained model. As a result of the analysis, situations that may decrease working efficiency are notified to the user in real-time to increase the user's well-being. In order for the study to be easily used by everyone, a user interface has been designed so that the user can customize it in every aspect. Improvements are made by making the designed interface used by people of different ages and professions.

#### **2 Related Works**

Object detection algorithms improved significantly with the help of AI [8]. Pose estimation is the detection of body parts in images or videos. Most studies [9–16] use a top-down approach for detection and estimation. As the first step in this approach, the person's part is detected, and then the body parts are estimated. Finally, pose estimation is made for each person. Although this approach applies to single-person detection, it cannot capture spatial dependencies between different people that require global inference. Studies [2] and [17] use a bottom-up approach in pose estimation. In this approach, all parts of each individual in the image are identified. Then, parts of different individuals are combined and grouped. Although this approach works slower than the top-down approach, it gives more accurate results because it can capture spatial dependencies between different people that require global inference [2]. Figure 1 presents an example of these approaches. For this reason, the bottom-up approach was preferred in this study.

**Fig. 1.** Top: typical top-down approach. Down: typical bottom-up approach [26, 27].

## **3 Methodology**

A real-time decision support system application is developed for the well-being of computer users. In the OpenPose [5] study, which has proven its success for the purpose, the pose estimation model was retrained using the transfer learning technique utilizing the customized MPII Human Pose [18] dataset.

#### **3.1 Environment**

The camera used for image transfer is the bisoncam nb pro, which is one of the models that comes internally in personal computers. The reason for choosing this model is to show that the system can work without any extra financial cost. This is a standard camera with a fixed focus, field of view of 60°, and maximum resolution of 720 p/30 fps. Nvidia Tesla T4 graphics card specially designed for AI applications was used to shorten the time for the model training phase. The high-end graphical processor has Turing architecture, 16 GB GDDR6 memory capacity, 320 Turing cores, and 2560 Cuda cores. For the testing phase of the pose estimation model, a system with similar characteristics to the average computer user was chosen. This system has Nvidia GTX 970 m graphics card, Intel I7-6700HQ 2.60 GHz central processor, and 16 GB DDR4L short-term memory.

#### **3.2 Dataset**

The pose detection model utilized the "MPII Human Pose" dataset via transfer learning. This dataset contains approximately 25000 images containing more than 40000 people with annotated body joints. Images were systematically collected using an established taxonomy of human activities each day. Overall, the dataset covers 410 human activities, and each image is provided with an activity tag [18]. In this study, only human exposure estimation was made without distinguishing activity.

#### **3.3 Model Development**

As with many bottom-up approaches, the proposed model first detects each person's parts (key points) in the image, then assigns the parts to different people. As in Fig. 2, the network is first used to extract features from the image using one of the models in studies [19–22] with its first few layers. A set of feature maps named F is created. Then F is fed into a network of two parallel convolutional layers. Stage-1 predicts 18 confidence maps, each representing a specific part of the human pose skeleton. At this stage, the network generates several Part Affinity Fields (PAFs) (1) L1 = F1(F), where F1 refers to the CNNs for inference in Stage-1. The predictions from the previous stage and the original image features F are combined and used to produce refined predictions at each subsequent stage.

Stage-2 predicts a value of 38 PAFs, which represents the degree of relationship between the parts. The successive stages are used to correct the forecasts made by each branch. Two-sided graphs were created between pairs using part confidence maps. By using PAFs values, weak links in bilateral charts are trimmed, and pose skeletons are predicted.

**Fig. 2.** Flowchart of OpenPose architecture [5].

$$L^{\mathfrak{t}} = \phi^{\mathfrak{t}}\left(F, L^{\mathfrak{t}-1}\right), \quad \forall \mathfrak{2} \le \mathfrak{r} \le \mathbf{T}\_{\mathbf{P}} \tag{1}$$

In Eq. (1), φτ represent the CNNs for inference at stage t, and Tp is the total number of PAF stages. After Tp iterations, the process is repeated to detect confidence maps, starting with the most recent PAF estimate.

$$\mathbf{S}^{\rm TP} = \boldsymbol{\rho}^{\rm r} \left( \mathbf{F}, \mathbf{L}^{\rm TP} \right), \quad \forall \mathbf{r} = \mathbf{T}\_{\rm P} \tag{2}$$

$$\mathbf{S}^{\mathsf{r}} = \rho^{\mathsf{r}}(\mathbf{F}, \mathbf{L}^{\mathsf{TP}}, \mathbf{S}^{\mathsf{r}-1}), \quad \mathsf{\forall} \mathbf{T}\_{\mathsf{P}} < \mathsf{r} \le = \mathsf{T}\_{\mathsf{P}} + \mathsf{T}\_{\mathsf{C}} \tag{3}$$

In Eq. (3), ρτ represents the number of CNNs for inference at stage t, and Tc is the number of total confidence map stages.

Confidence map results are estimated above the most recent processed PAF estimates, making a barely noticeable difference in confidence map stages. A loss function is applied at the end of each stage to direct the network to iteratively estimate the PAF values of the body parts in the first stage and the confidence maps in the second stage. L2 loss is used between the expected forecasts and the baseline information maps and fields.

$$\left\| f\_L^{l\_l} = \sum\_{c=1}^{C} \sum\_{p} W(p) . \right\| L\_c^{l\_l}(p) - L\_c(p) \left\| \right\|\_{2'}^2 \tag{4}$$

$$\|f\_S^{\mathbb{N}} = \sum\_{j=1}^{J} \sum\_{p} W(p) . \left\| S\_c^{\mathbb{N}}(p) - S\_{\bar{l}}(p) \right\|\_{2'}^2 \tag{5}$$

Equation (5) has spatially weighted the loss functions to address a practical problem where some datasets do not thoroughly label all people. Where Eq. (4) Lc is the PAF actual reference value (ground truth), Sj is the true confidence map, and is a binary mask with W(p) = 0 when p is missing annotation on the pixel. The mask is used during training to avoid penalizing true positive guesses. Interim inspection at each stage fills the gradient periodically, eliminating the vanishing gradient problem.

To increase the efficiency of the study, it was foreseen that it would be sufficient to determine only the eyes, ears, nose, shoulders, and neck ending to be used in the study. In order to detect only these body parts, a new data set consisting of the necessary body parts in the data set was created. With this dataset, the model was retrained using the transfer learning method to benefit the pre-trained weights of the MobilNetV2 [22].

For conditions that may pose a risk of disease, the angle values between the limbs were taken as a reference [6]. The reference values can be changed by ±30% with the slide bar in the program's interface and by ±250% in the advanced settings menu. When the values calculated due to exposure estimation went out of the reference value limits, the system took 50 samples at equal intervals for 10 s, and the average was calculated. The calculated value and the reference values are compared. When there is a situation that may pose a disease risk, the user is notified in real-time with notifications as in Fig. 3.

**Fig. 3.** Notification example of work in Windows 10 environment

#### **3.3.1 Creation of the User Interface**

The target audience of this study is computer users, and the user interface has been designed so that everyone can use it easily. In addition, the work has been made saved as a file with a.exe extension. The designed interface is designed in such a way that the user can customize it in every aspect, as seen in Fig. 4. The interface's main features are the selection of the posture tests, customization of wrong posture reference values, and customization of notification frequency. The settings also have five different profiles to save the made customizations.

**Fig. 4.** Improved user interface

## **4 Results and Discussions**

Using the images reserved from the dataset for testing, the high performed models, i.e., PersonLab [23], METU [24], Associative Emb. [25], and OpenPose [5] models were compared with the proposed method in Table 1 with respect to accuracy and precision. Since the most critical parameter for the applicability of the study is speed, the proposed method performed better than other methods.

**Table 1.** Performance comparison of the proposed method and the methods in the literature.

## **5 Conclusion**

Health is one of the significant elements that reduce efficiency in organizations. In addition, one of the most critical expense items in most countries is healthcare. This study aimed to contribute to the country's economy by identifying situations that reduce working efficiency and quality of life. In order to achieve this aim, an optimized pose estimation model is proposed using the transfer learning technique. With the developed application, posture disorder analysis will be performed, and health problems that may occur in the waist, neck, and joint regions will be prevented. Visual disturbances will be prevented by analyzing the distance to the monitor, working environment lighting, and usage time. With the help of the proposed model, the work environment will be analyzed dynamically to support healthy working conditions.

## **References**

**Open Access** This chapter is licensed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

The images or other third party material in this chapter are included in the chapter's Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the chapter's Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.

## **On Two Approaches to Clustering of Cross-Sections of Rotationally Symmetric Objects**

Paulina Baczy´nska1(B), Agnieszka Kaliszewska<sup>1</sup>, and Monika Syga<sup>2</sup>

<sup>1</sup> System Research Institute, Polish Academy of Sciences, ul. Newelska 6, 01-447 Warszawa, Poland<EMAIL_ADDRESS><sup>2</sup> Warsaw University of Technology, Faculty of Mathematics and Information Science, Koszykowa 75, 00-662 Warsaw, Poland<EMAIL_ADDRESS>
**Abstract.** We analyze two approaches to clustering 2D shapes representing cross-sections of rotationally symmetrical objects. These approaches are based on two ways of shape representation - contours and silhouettes - and a number of similarity measures which are based on a combination of Procrustes analysis (PA) and Dynamic Time Warping (DTW) as well as on binary matrix analysis. The comparison of efficiency of the proposed approaches is performed on datasets of archaeological ceramic vessels.

**Keywords:** Clustering · Shape analysis · Procrustes analysis · Dynamic time warping · Similarity measures · Contours · Binary matrix comparison · Hamming distance · Vari distance · Rogers-Tanimoto distance · Archaeological ceramic vessels

## **1 Introduction and Problem Formulation**

Clustering methods, which rely on grouping data sets into clusters without imposing the number of clusters in advance [16,20], are widely used in many applications from natural speech recognition to biology and medicine (e.g. [13]). In order to obtain significant results of clustering, aside from the choice of the specific clustering algorithm, the following issues are of fundamental importance:

The problem of suitable data representation has been recognized in the literature, see e.g. [6]. The choice of data representation for the investigated data set has a considerable impact on the possibility to achieve satisfying results. Studies on the impact of shape representation on clustering architectural plans have been conducted in e.g. [17].

#### **1.1 The Problem**

Our contribution is devoted to clustering 2D objects with respect to shape and size. We consider two types of shape representations: contours and silhouettes. For contour representations we use similarity measures based on a combination of DTW [1,4,15] and PA [5,9,10], as discussed in [12]. For silhouettes representations we use a number of binary distances, e.g. Hamming, Vari, Rogers-Tanimoto [18]) as well as Pattern-difference, Size-difference distance [3].

In the case of contour representation, the overall similarity measures are defined as combinations of size and shape similarity measures as proposed in [12], see Sect. 3, i.e. the clustering is performed simultaneously with respect to size and shape, whereas in the case of silhouette shape representation, we start with the clustering with respect to size, and, within the obtained (size) clusters, the clustering is next performed with respect to shape (Sect. 2).

#### **1.2 Motivation and Contribution**

The aim of the present contribution is twofold. First, by using binary matrix representation of the investigated 2D shapes we propose a clustering algorithm based on a number of discrete similarity measures combined with Affinity Propagation algorithm. Secondly, we investigate the impact of the choice of shape representation and similarity measures on the results of clustering of 2D crosssections of rotationally symmetric 3D objects, by comparing the binary matrix approach with the contour-based approach proposed in [12].

Both approaches, the silhouette-based approach proposed in Sect. 2 and the contour-based approach proposed in [12] and recalled briefly in Sect. 3 can be applied to clustering of general 2D objects. Our motivation comes from digital cultural heritage and is related to clustering of archaeological ceramic vessels with respect to size and shape. Due to technological principles, ceramic vessels are rotationally symmetric, and so the clustering can be limited to their 2D cross sections (see Fig. 1). The papers devoted to this topic are scarce in the literature, e.g. [8,11,21].

The experiments are performed on real-life archaeological data representing cross-sections of ceramic vessels. Our entry data are in the form of black and white images of a unified resolution. When clustering archaeological ceramic data one can hardly specify the number of clusters in advance, so we are bound to use algorithms which do not require the number of clusters to be predefined.

The representations and the similarity measures, are discussed in Sect. 2 and Sect. 3. The results of experiments for archaeological ceramic data sets and the discussion are presented in Sect. 4.

**Fig. 1.** The transformation of a section of rotationally symmetric object into input data.

#### **2 Silhouette-based Approach**

#### **2.1 Data Representation**

In this approach we represent shapes as matrices of pixels, to be clustered as binary matrices or, in some cases, as binary vectors. Process of preparing input data consists of two steps.

The first step relies on the analysis of the size of original input data. Size analysis is performed by KMeans cluster algorithm. To find the optimal division of the dataset, this algorithm finds clusters consisting of similar-sized objects. In the case of our experimental data, the typical outcome of KMeans algorithm, is one or two subsets-clusters of objects of similar size. This step is performed by an application written in Python 3.6. in order to retain the information about size of the investigated objects, as in the second step we perform normalization, which leads to the loss of size information. This information is recalled in the last step, before the final clustering.

Second step is related to data transformation. We start with transforming every object to the chosen fixed size, which is related to resolution of provided objects. Next step is to create matrix representation of each object. The rows and columns of the matrix represent pixels of objects. Knowing that every image consists of black and white pixels only, the matrix of pixels is created as follows: element (i, j) is given as 1, when the particular pixel is black and 0 when the particular pixel is white. This matrix represent the objects. This step is performed by application written in Python 3.6.

After data preparation, we perform clustering, based on similarity measures dedicated to binary data [3]. In this process we recall the size information (division obtained through KMeans), to be included in the final clustering result. Depending on the number of clusters obtained by KMeans (clustering with respect to size), next step, i.e. clustering with respect to shape is performed independently, i.e. in each cluster separately.

#### **2.2 Similarity Measure**

A lot of measures have been proposed for computing the similarity/dissimilarity between two vectors. In the present investigation we are interested only in those dedicated to binary input data. We base our choice on 76 binary similarity and dissimilarity measures recalled in [3]. Our selection of similarity measures for further analysis was focused on the diversity. As a result, five measures were selected for further analysis. Let X, Y - binary matrices in the same sizes. Let xij ∈ X, yij ∈ Y be elements located at the same positions in rows and columns of matrices X and Y , respectively. Let dxy be the number of instances where specific combinations of zero and ones occur: d00, d11, d01, d10. Initially, every dxy value is set to 0. Next, we update them, i.e. for every pair of elements xij and yij the following operations are performed:

In our clustering analysis we use the following similarity measures/distances:

Several characteristic features of the above similarity measures are observed:

After calculating the similarity matrix, the effectiveness of each of the similarity measures was evaluated by the clustering method. The clustering is performed by the Affinity Propagation algorithm (see e.g. [7]), which does not require a predefined number of clusters. This is an essential feature of our archaeological application. This part of clustering process is performed by application written in Python 3.6.

Below there are main algorithms used in analysis. Symbols used in algorithms:

1. In[x<sup>I</sup>*<sup>n</sup>* , y<sup>I</sup>*<sup>n</sup>* ] - original image I<sup>n</sup> with sizes: x<sup>I</sup>*<sup>n</sup>* - width, y<sup>I</sup>*<sup>n</sup>* - height

#### **Algorithm 1.** *Single image preparation*

#### **Algorithm 2.** *Size clustering*

*1: Start size analysis: 2: for* <sup>i</sup> = 0;i<N;<sup>i</sup> + + *do 3: for* <sup>j</sup> = 0; j<N; <sup>j</sup> + + *do 4: if* (xAi · <sup>y</sup>Ai) <sup>÷</sup> (xAj · <sup>y</sup>Aj ) > D *then 5: do KMeans algorithm with given number of clusters 6: end if 7: end for 8: end for*

#### **Algorithm 3.** getInstances(Ba, Bb)

*1: Set variables:* <sup>d</sup><sup>00</sup> = 0*,* <sup>d</sup><sup>11</sup> = 0*,* <sup>d</sup><sup>01</sup> = 0*,* <sup>d</sup><sup>10</sup> = 0 *2: for* <sup>i</sup> = 0;i<x<sup>B</sup>*<sup>a</sup>* ;<sup>i</sup> + + *do 3: for* <sup>j</sup> = 0; j<y<sup>B</sup>*<sup>a</sup>* ; <sup>j</sup> + + *do 4: if* <sup>B</sup>a(i, j)=0 <sup>∧</sup> <sup>B</sup>b(i, j)=0 *then 5:* <sup>d</sup><sup>00</sup> <sup>=</sup> <sup>d</sup><sup>00</sup> + 1 *6: else if* <sup>B</sup>a(i, j)=1 <sup>∧</sup> <sup>B</sup>b(i, j)=1 *then 7:* <sup>d</sup><sup>11</sup> <sup>=</sup> <sup>d</sup><sup>11</sup> + 1 *8: else if* <sup>B</sup>a(i, j)=0 <sup>∧</sup> <sup>B</sup>b(i, j)=1 *then 9:* <sup>d</sup><sup>01</sup> <sup>=</sup> <sup>d</sup><sup>01</sup> + 1 *10: else if* <sup>B</sup>a(i, j)=1 <sup>∧</sup> <sup>B</sup>b(i, j)=0 *then 11:* <sup>d</sup><sup>10</sup> <sup>=</sup> <sup>d</sup><sup>10</sup> + 1

*12: end if 13: end for 14: end for 15: return* <sup>d</sup>00*,* <sup>d</sup>11*,* <sup>d</sup>01*,* <sup>d</sup><sup>10</sup>

**Algorithm 4.** *Similarity measures*

*1: for* <sup>i</sup> = 0;i<N;<sup>i</sup> + + *do 2: for* <sup>j</sup> = 0; j<N; <sup>j</sup> + + *do 3:* getInstances(Bi, B<sup>j</sup> ) *4: Hamming distance:* <sup>M</sup><sup>H</sup>*i,j* (Bi, B<sup>j</sup> ) *5: Vari distance:* <sup>M</sup><sup>V</sup>*i,j* (Bi, B<sup>j</sup> ) *6: Rogers - Tanimoto dissimilarity coefficient:* <sup>M</sup>RT*i,j* (Bi, B<sup>j</sup> ) *7: Pattern - Difference distance:* <sup>M</sup>PD*i,j* (Bi, B<sup>j</sup> ) *8: Size - Difference distance:* <sup>M</sup>SD*i,j* (Bi, B<sup>j</sup> )

*9: end for*

## *10: end for*

#### **Algorithm 5.** *Affinity propagation*

An important feature of the approach proposed in Algorithm 1-Algorithm 5 is that the clustering analysis is performed sequentially: first with respect to size ( K-Means algorithm), and next with respect to shape (Affinity Propagation algorithm).

## **3 Contour-based Approach**

Below we recall the contour based approach, as proposed in [12]. Contours (boundary discrete curves) are extracted from cross-sections by standard contour extraction techniques and smoothed by Savitzky-Golay filtering. Hence, objects to be clustered, are discrete curves α, i.e. pairs of vectors,

$$\alpha := (x(i), y(i)), \ i = 1, \ldots, k\_{\alpha}. \tag{1}$$

#### **3.1 Composite DTW-PA Similarity Measures**

By n ≥ 2 we denote the cardinality of the set of contours to be clustered. By i and j, i, j = 1, ..., n, we denote the i-th and the j-th contour, respectively. We calculate ,,distances" between contours i and j, i = j by using the formulas (2), (3), (4) which measure the degree of similarity between i and j,

$$\text{Procrustes measure: } pa(i, j) := \max\{PA(i, j), PA(j, i)\}, \tag{2}$$

where P A(i, j) denotes the Procrustes measure between i and j obtained from Matlab Statistics and Machine Learning Toolbox (for more details see [12]),

$$c \text{ direct composition measure: } dc(i, j) := \max\{DC(i, j), DC(j, i)\}, \qquad (3)$$

where DC(i, j) = DTW(i, Z<sup>j</sup> ), DTW measure is calculated in Matlab Signal Processing Toolbox, and Z<sup>j</sup> is the optimal curve obtained from P A(i, j), the measure DC was introduced and analysed in [12],

$$\text{scale component measure: } \gamma(i, j) := 1 - \min\{\gamma^\*(i, j), \gamma^\*(j, i)\},\tag{4}$$

where γ∗(i, j) is the optimal scaling factor obtained from the P A(i, j) (see [12] for more details). We put pa(i, i)=0, dc(i, i)=0, γ(i, i)=0. To ensure comparability of the above defined measures, we use the following normalization formulas for dc and γ

$$ndc(j) := \frac{1}{\max\_{i=1,\ldots,n} dc(i,j)}, \quad \text{for} \quad j = 1,\ldots,n.$$

$$n\gamma(j) := \frac{1}{\max\_{i=1,\ldots,n} \gamma(i,j)}, \quad \text{for} \quad j = 1,\ldots,n.$$

In absence of normalization we put ndc(j) = ndc = 1, for all j = 1, ..., n or nγ(j) = nγ = 1, for all j = 1, ..., n.

#### **3.2 Similarity Matrix**

Now we recall the similarity measure (5) and the similarity matrix (6), which where defined and investigated in [12]. Let μ, λ, ω <sup>∈</sup> <sup>R</sup> be given numbers (weights). The similarity measure (SM) is given as follows:

$$SM\_{ij}(\mu,\lambda,\omega,ndc,n\gamma) := \tag{5}$$

$$\mu \cdot pa(i,j) + \lambda \cdot ndc(j) \cdot dc(i,j) + \omega \cdot n\gamma(j) \cdot \gamma(i,j).$$

Depending upon the choice of parameters μ, λ, ω <sup>∈</sup> <sup>R</sup> we obtain basic measures pa(i, j), dc(i, j), γ(i, j), combinations of two or three of them, with, or without, normalization. By calculating SMij according to (5) for every pair of contours i and j, i, j = 1, ..., n, we get the similarity matrix

$$SM(\mu, \lambda, \omega, ndc, n\gamma),\tag{6}$$

which is symmetric and has zeros on the main diagonal. Different choices of μ, λ, ω, ndc, nγ lead to different similarity matrices and consequently, to different clustering outcomes. In particular, we obtain

1. weighted Procrustes and scale component matrix

$$WPSM = SM(\mu, 0, \omega, 1, 1),\tag{7}$$

2. weighted direct composition and scale component matrix where direct composition values are normalized

$$WNDCSM = SM(0, \lambda, \omega, ndc, 1),\tag{8}$$

3. weighted direct composition and scale component matrix, where both direct composition values and scale component values are normalized

$$WNDCNSM = SM(0, \lambda, \omega, ndc, n\gamma). \tag{9}$$

We perform clustering on the basis of the similarity matrix (6) by the standard hierarchical algorithms (Matlab Statistics and Machine Learning Toolbox) and generate the dendrogram. The average linkage is used to measure the distance between clusters.

An important feature of the contour-based approach is that the weights appearing in the definition of the similarity measures (5) allow the simultaneous clustering with respect to size and shape.

#### **4 Discussion of the Results of the Experiment**

Below we summarize the clustering results obtained with the help of methods from Sect. 2 and Sect. 3. The experiment has been conducted on seven sets of reallife data compiled from archaeological material. One of the datasets is presented in Fig. 2. It is worth noting that there are no benchmark data sets related to clustering of archaeological ceramics.

**Fig. 2.** Set 1

According to general basic evaluation criteria (e.g. [16]), clustering results are acceptable when the resulting clusters are well defined, i.e. the distances between elements within the cluster are small (intra clusters characteristic), while the distances between clusters are high (extra clusters characteristic). Due to the unlabeled nature of the investigated data our evaluation is expert-based.

The percentage of correctly clustered elements is summarized in Table 1. Average scores have been calculated over all sets for all methods. For readability, best scores for both classes of methods (i.e. contour- and silhouette-based) are marked in bold.

Based on the results, we feel confident to evaluate the performance of both approaches as acceptable, with the average scores for each method between 78.68%–86.63%.