Paper Title: Towards End-to-End Explainable Facial Action Unit Recognition via Vision-Language Joint Learning.



















University of Glasgow
GlasgowUnited Kingdom
x.ge.2@research.gla.ac.uk

 

  University of GlasgowGlasgowUnited Kingdom
j.fu.3@research.gla.ac.uk



Fuzhou University
FuzhouChina
chenfuhai3c@163.com



Tianjin University
TianjinChina
anshan.tju@gmail.com



University of Trento
TrentoItaly
niculae.sebe@unitn.it


University of GlasgowGlasgowUnited Kingdom
joemon.jose@glasgow.ac.uk






^* Equal contribution. † Corresponding author. 





§ ABSTRACT 
Facial action units (AUs), as defined in the Facial Action Coding System (FACS), have received significant research interest owing to their diverse range of applications in facial state analysis. Current mainstream FAU recognition models have a notable limitation, i.e., focusing only on the accuracy of AU recognition and overlooking explanations of corresponding AU states. In this paper, we propose an end-to-end Vision-Language joint learning network for explainable FAU recognition (termed VL-FAU), which aims to reinforce AU representation capability and language interpretability through the integration of joint multimodal tasks. Specifically, VL-FAU brings together language models to generate fine-grained local muscle descriptions and distinguishable global face description when optimising FAU recognition. Through this, the global facial representation and its local AU representations will achieve higher distinguishability among different AUs and different subjects. In addition, multi-level AU representation learning is utilised to improve AU individual attention-aware representation capabilities based on multi-scale combined facial stem feature. Extensive experiments on DISFA and BP4D AU datasets show that the proposed approach achieves superior performance over the state-of-the-art methods on most of the metrics. In addition, compared with mainstream FAU recognition methods, VL-FAU can provide local- and global-level interpretability language descriptions with the AUs' predictions. 







<ccs2012>
   <concept>
       <concept_id>10010147.10010178.10010224.10010240.10010241</concept_id>
       <concept_desc>Computing methodologies Image representations</concept_desc>
       <concept_significance>100</concept_significance>
       </concept>
   <concept>
       <concept_id>10010147.10010178.10010224.10010225.10003479</concept_id>
       <concept_desc>Computing methodologies Biometrics</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>


[100]Computing methodologies Image representations
[500]Computing methodologies Biometrics
























Towards End-to-End Explainable Facial Action Unit Recognition via Vision-Language Joint Learning 
    Joemon M. Jose
    August 5, 2024
=================================================================================================






§ INTRODUCTION

    Recognized as a fundamental research challenge by the Facial Action Coding System (FACS) <cit.>, facial action unit (FAU) recognition holds significance for analyzing facial states, including facial expression analysis <cit.>, diagnosing mental health issues <cit.>, detecting deception <cit.>, and more. As a result, this area of study has gained increasing interest in recent years. 

    Early works <cit.> design hand-crafted features based on the inherent characteristics of facial muscle movements to determine FAU states. However, hand-crafted shallow features prove inadequate to address the major challenges of FAU recognition, i.e. identifying subtle facial changes induced by different AUs, as well as variations stemming from individual physiology. 
    Instead, deep learning based studies <cit.> have recently been explored to improve the AU representations as a multi-label classification problem, where the AU individual characteristics are adequately learned to boost the FAU recognition. 
    Therefore, improving the individual AU  representation becomes the key to boosting the FAU recognition task. The main challenge is how to maintain discriminative inter-AU features while ensuring rich semantic representation capabilities within AUs.


 
    
 

    

  
    To enhance the individual representation of each AU, mainstream studies <cit.> adopt a multi-branch AU recognition framework with independent classifiers to decouple different AUs.
    For instance, JAA-Net <cit.> proposes a face alignment network that predefines local muscle regions for various AUs using detected landmarks. It then refines all AU region representations through multiple independent convolutional networks dedicated to different AU classifications.
    The predetermined AU local regions and their feature representations establish explicit and distinct branches for conveying information essential for recognizing different AU states. However, they overlook the pertinent information originating from undefined facial areas. Therefore, 
    ARR <cit.> is proposed to directly leverage a shared global face feature to explore the local AU representations by subsequent independent convolution blocks in different AU branches, covering the whole face information. 
    However, recent studies <cit.> suggest that the above methods fail to capture intrinsic relationships between independent AU branches, which are actually undeniable <cit.>. 
    To this end, some works <cit.> propose to build relationships between predefined AU branches by prior relationship statistics in datasets or implicit relationship reasoning in the modelling networks, such as Long Short-Term Memory network (LSTM) <cit.>, graph convolutional network (GCN), etc. 
    

     
    
    Despite such progress, two prevalent defects remain unexplored: (i) improving the representation capability of individual AUs through inter-AU relational connections may compromise the distinctiveness of features among AUs, and (ii) directly obtaining classification results lacks an explainable basis for judgments. 
    The main reason for the former is that inter-AU information  based relationship reasoning networks, such as GCN <cit.> or GNN <cit.> etc., are prone to the over-smoothing problem <cit.> among the different AU nodes, especially on small-scale face datasets. This makes differentiating the AU states difficult. 
    The latter issue is caused by the current mainstream paradigm as shown in Figure <ref>, which only focuses on the classification results and ignores the corresponding explanations.

    
    In this work, we bring fresh insights toward explainable facial AU recognition via the integration of language generation supervisors within an end-to-end FAU recognition network. 
    We argue that detailed AU language descriptions, such as “The corners of the lips are markedly raised and angled up obliquely. The nasolabial furrow has deepened ..." for activated AU12 (Lip Corner Puller) in Figure <ref>, can provide more linguistic semantics and relations to corresponding AU appearance features than mainstream relational reasoning methods <cit.>. Furthermore, different AUs correspond to different descriptions, keeping them better distinguishable. 
    Different from encoding pre-provided activated AU language descriptions into AU classification in <cit.>, we introduce the language generation model as language semantic supervision for the classification of different AU states. It can provide AU prediction with the explainable language descriptions as Figure <ref> while providing sufficient semantic to enrich AU representations and supervising the AU detection pertinently to distinguish AU.
    

    Motivated by the above insights, we propose a novel end-to-end vision-language joint learning model (termed VL-FAU) for FAU detection with explainable language generation. 
    Compared with the mainstream methods in Figure <ref>, our main innovations lie in two aspects: (i) exploring the potential of joint language generation auxiliary training for intra-AU semantic enhancement and inter-AU semantic feature differentiation 
    and (ii) providing interpretability for end-to-end FAU recognition. 
    Specifically, we first introduce a new dual-level AU feature refinement based on multi-scale combined representations from a vision backbone. This way, we provide each AU branch with a unique attention-aware representation ability.
    Secondly, to improve the semantics and guarantee the distinguishability of AU features as well as their interpretability, we integrate the end-to-end multi-branch framework with the local and global language generations for explicit semantic guidance. 
    On one hand, each local branch of AU recognition is simultaneously supervised by a corresponding local language generator via fine-grained semantic-supervised optimization.
    Such schema constrains the semantics and relationships of AU features by local language modeling and improves the inter-branch distinguishability for each face image. 
    On the other hand, global facial features within and between subjects are difficult to distinguish because muscle changes are mostly subtle under different states. 
    
    To this end, we introduce a global language model to generate a description focusing on all activated muscles as global semantic supervision. 
    It provides better distinction between different whole-face representations within and between subjects via multiple facial state foci as shown in Figure <ref>. 
    Finally, vision AU recognition and language description generation are jointly optimized. 
    

    
    The contributions of this work are as follows: 
    
    
    
  * We propose a novel end-to-end vision-language joint learning scheme for explainable FAU recognition (VL-FAU),  which leverages auxiliary training of language generators to improve discrimination and explanation for FAU recognition.
    
    
    
  * We design a new dual-level AU representation learning method based on the multi-scaled facial representation for AU branches, which provides stronger attention-aware AU representation ability;
    
    
  * We design a novel joint supervision method with local and global language generations for FAU recognition. In such schema, the local language generation provides explicit semantic supervision of each independent AU branch, thereby improving inter-AU discriminability and intra-AU detailed semantics, while the global language model maintains the global distinguishability of different facial states within and between subjects. 
    
  * We extend FAU datasets with new local and global language descriptions for different facial muscle states to facilitate language-interpretable FAU recognition. 
    
    
    We conduct extensive experiments on two widely used benchmarks, i.e., BP4D and DISFA, to evaluate the proposed VL-FAU model. VL-FAU outperforms state-of-the-art approaches in AU recognition and provides detailed language descriptions of the individual AU decision and global face state for explanations.
    




§ RELATED WORK

    
    Facial AU Recognition.
    The facial AU recognition task is popularly treated as a multi-label classification problem, utilizing multi-branch independent AU classifiers to predict activation states. 
    From this perspective, most existing methods can be roughly categorized into two groups:
    region-localization based AU recognition <cit.> and global-refinement based AU recognition <cit.>.
    The former methods represent precise AU  representations by local patch feature learning and maintain certain differences between different AU features. 
    For instance, in some early works <cit.>, there is a need to predefine the patch location first. 
    Recent JAA-Net <cit.> joined AU recognition and face alignment in a unified framework, employing detected landmarks to locate specific AU regions.   
    However, all the above methods focused only on independent regions, without considering non-defined face areas and AU correlations. 
    The latter global-refinement based methods extract AU branch features by directly refining the global face representation to ensure that useful information is not lost. 
    For example, <cit.> proposed a transformer-based multi-branch AU recognition network, where each branch leverages a shared global face representation and refines it by independent convolution layers. 
    Similarly, the correlations between AUs are also ignored.
    Recent works <cit.> pay attention to capturing the relations among AUs for modeling the inherent muscle linkages, supplying information transfers between AU branches.  
    For instance, <cit.> utilized multiple GCNs to model AU relations, requiring additional pre-statistical AU correlation references from the training set.
    However, this inter-branch information interaction somewhat breaks the original categorical independence between branches, which in turn weakens the discriminability of AU individuals. Moreover, all the above methods are flawed regarding explainable FAU recognition.


    Multi-task Joint Learning. 
    Recently, multi-task joint learning <cit.> has been an emerging topic in deep learning research, which enables the same target to efficiently obtain multiple representation capabilities in a unified model.
    For example, <cit.> joined face alignment into a facial AU recognition framework to assist in locating the muscles corresponding to AUs. 
    <cit.> combined four face tasks into a unified framework and assigned an attention module for each face analysis subnet. 
    However, these methods only focus on computer vision tasks and ignore the modality complementary advantages of multiple modality-task joint learning, i.e. interpretability complementarity and semantic complementarity.
    For example, <cit.> proposed a joint multi-modal aspect-sentiment analysis with auxiliary cross-modal relation detection, which can provide image sentiment predictions with explicit language explanations. 
    However, most visual-language joint learning models <cit.> rely on the joint input of both modalities during inference process. 

    
    In this paper, we argue that explicit language can facilitate the modeling of inter-AU relevance and diversity.
    Each AU description can include muscle details and relationship descriptions identified by AU, which not only replace the visual relational reasoning among AUs, but also ensure the independence of AUs to improve distinguishability.   
    The most relevant research to ours is SEV-Net <cit.>, which introduced AU language descriptions as a prior embedding into AU representations. However, significant drawbacks are that SEV-Net relied on the human pre-given language descriptions of activated AUs for inference and followed the mainstream AU recognition paradigm in Figure <ref> for only AU prediction.
    In contrast to <cit.>, our VL-FAU introduces language generation auxiliary models for local AU prediction and global face refinement. It focuses on end-to-end language-explainable facial-image AU recognition without any pre-given reference for inference. 
    
        


§ APPROACH

    As shown in Fig. <ref>, the proposed approach –VL-FAU– consists of two key components, i.e., multi-level AU representation learning, and local and global auxiliary language generation. The former contains dual-level AU individual refinement for multi-branch FAU recognition based on multi-scale feature combinations from Swin-Transformer. 
    The latter consists of local AU language generation, which facilitates semantic supervision of each AU  and global facial language generation for whole-face semantic supervision.
    Both of these can help AU representations become more robust and distinguishable, thus improving the performance. 
    Thus, we create an end-to-end framework with a multi-label classifier for explainable FAU recognition, supervised with local and global AU language generation auxiliaries.


    

 §.§ Multi-level AU Representation Learning
 
        

  §.§.§ Global Facial Representation Extraction
  
        Given a face image I, we adapt the widely-used Swin-Transformer as the stem network <cit.> to extract the global feature V by combining the multi-scale representations from different stages. 
        Multi-scale feature learning and combination <cit.> is a popular image representation approach to leverage different levels of semantics from the backbone, where the low-level features from shallow blocks 
       contain more texture information and the deeper blocks contain high-level semantics. 
        In the work, we follow this strategy (multi-scale combination – MSC) to represent the global face image, which contains 4 independent 3×3 convolutions to learn and reshape the representations (V^s_1, V^s_2, V^s_3, V^s_4) from different stem stages. 
        After that, four-level feature maps are combined as the global facial representation V by a learnable Linear layer.
        The above process can be formulated as:
        
    V=W^m([ Conv(V^s_1): Conv(V^s_2): Conv(V^s_3): Conv(V^s_4)]),
   
        where [:,:] means the concatenation operation, W^m∈ℝ^D× d is mapping parameter and Conv means 3×3 convolution.
        For simplicity, we will not repeat the detailed structure of the stem Swin-Transformer network <cit.> here.

    

  §.§.§ Dual-level AU Individual Refinement
 
        Multi-branch network is a good way to learn the rich and fine-grained individual facial AU representation for the final classification. 
        In this paper, we propose a new multi-branch dual-level AU individual refinement (termed DAIR) for the final attention-aware AU representations.
        As shown at the top of Figure <ref>, each DAIR in each branch contains two levels of attention learning, i.e. channel-level and spatial-level, which employs two pooling strategies <cit.>. 
        While existing research from both perspectives is extensive <cit.>, this study represents the first adaptation within independent FAU branches for enhancing the independent fine-grained  attention-aware AU representations rather than coarse-grained global representation. 
        Specifically, following <cit.>, we extract the max-pooled and average-pooled channel-level vectors (F^c_max and F^c_avg) along the spatial axis for channel-wise attention mining. After that, two shared learnable multi-layer perceptrons (MLP) are used to obtain the mapping of channel-wise vectors and then a sigmoid function is applied to get the i-th individual AU-channel attention. 
        Similarly, we later extract the max-pooled and average-pooled spatial-level vectors (F^s_max and F^s_avg) along the channel axis for spatial-wise attention mining.
        We utilise a 3×3 convolution to generate a spatial attention map, which focuses on highly responsive muscle areas associated with the current AU.
        Finally, the i-th individual AU attention-aware representation V̂_i can be obtained as below:
        
    V̅_i =  σ( MLP(F^c_max) +  MLP(F^c_avg)) V,
   
        
    V̂_i =  Conv([F^s_max:F^s_avg]) V̅_i ,
   
        where σ is the sigmoid function. 

         

 §.§ Auxiliary Supervision with Local and Global Language Generation
 
         In addition to improving AU individual representations in multiple AU branches, our main innovation is to provide a new approach and inspiration for explainable FAU 
         
         recognition, i.e., joint auxiliary language generation for FAU recognition, rather than a new complex language model.  
         Language generation provides explicit semantic supervision while giving linguistic interpretability of the corresponding identified AUs, rather than just simply recognising the AUs. 
         It contains two aspects: (i) global face language generation for explicit semantic auxiliary supervision of the whole face image representation, and (ii) local AU language generation for individual AU semantic auxiliary supervision. 
         
        

  §.§.§ Global Language Generation 
 
        Different from the mainstream facial AU recognition <cit.>, we introduce a new global language auxiliary model to generate the language description for activated AUs of the whole face. 
        It brings benefits for subsequent multi-branch FAU recognition, i.e., it maintains the accuracy and variability of intra- and inter-subject stem features by explicitly focusing on multiple different AU muscles.
        Thus, it somewhat overcomes the data imbalance, i.e., inactive AUs are far more numerous than activated AUs in benchmarks.

        
        Specifically, as shown in Figure <ref>, we treat the multi-scale stem-feature V as the encoded image feature and input it into an attention-aware language decoder similar to image captioning <cit.>, which contains a soft-attention module to mine the attention-aware visual representation based on the past generated word s^g_0:i-1 for new word s^g_i. 
        For example, when generating current word s^g_i, we first calculate the soft-attention α_i between the image feature V and the last hidden state h_i-1 of generated word s^g_i-1 by linear-based mapping operations with SoftMax function <cit.>. And then the i-th attention-aware visual feature α_i V is combined with the last hidden state h_i-1, which is fed into the i-th LSTM cell <cit.> with the previous cell state c_i-1 to generate the new hidden state h_i and cell state c_i of new word. The above process can be formulated as:
        
    α_i =  SoftMax(W^a( ReLU(W^vV + W^h h_i-1))),
  
        
    (h_i, c_i) =  Cell([α_i V : h_i-1], c_i-1),
  
        where W^v∈ℝ^d× d, W^h ∈ℝ^h× d, and W^a∈ℝ^1× d are the parameters of mapping function. During the training process, we use a shared learnable parameter W^s∈ℝ^h× voc to obtain vocabulary-length predicted vector and obtain the max-score index as the predicted word, as follows:
        
    s_i^g =  arg max(W^s h_i),
  

        During inference process, the beam search <cit.> can be used to obtain the most optimal global description S^g = [s^g_1, ..., s^g_T]. 
        



        

  §.§.§ Local Language Generation.
 
        We introduce an individual local language generation model for each AU branch as an auxiliary semantic supervision, which can generate the corresponding fine-grained language description for each AU determination. 
        Compared with the global facial description, the local AU language description contains more facial muscle details described for corresponding AUs. 
        Besides, different AUs have different local language descriptions, which are more fine-grained and diverse. 
        The main motivation of the joint language model in each AU branch is not only to improve the distinguishability between AUs using fine-grained semantic auxiliary supervision, but also to provide specific language interpretation for each AU prediction.
        
        In particular, different from global language generation, each local language model utilizes the proposed DAIR to further refine the encoded face feature V, obtaining the distinguishable attention-aware AU representation V̂_i for the subsequent language generation. 
        The decoder architecture of each local language generation model is the same as the global language model. To save space, we use ∮_i to represent the i-th local language model for the i-th AU branch and omit the model details. Finally, we can obtain the i-th local AU description S^l_i = [s^l_1;i, ..., s^l_T;i] with length T, as follows:
         
    S^l_i = ∮_i( DAIR_i(V)),
  
        Note that, the local language models among the multiple AU branches are shared to maintain efficiency.
       
    

        
        
        

 §.§ Vision-Language Joint Learning

        VL-FAU joints facial AU recognition (vision) and description generation (language) into an end-to-end multi-branch network. Among them, FAU recognition is predominant due to the explicit AU annotations, while the language models are used for auxiliary semantic supervision and to improve feature diversity and distinguishability of visual stem and sub-branches. 
        For facial AU recognition, one fully connected layer with SoftMax function is used as a multi-label binary classifier to classify the AU activation state, which adopts a weighted multi-label cross-entropy loss function as follows,
        
    ℒ_Fau = - 1/N∑_i=1^Nγ_i [y_i  log (p(y_i)) + (1-y_i)  log (1-p(y_i)],
 
        
    γ_i = 1/ϵ_i/∑_i=1^N (1/ϵ_i)

        where N is AU number, y_i and p(y_i) denote the ground-truth and predicted probability for the i-th AU occurrence, respectively. 
         γ_i is a balancing weight of the i-th AU  calculated by the i-th AU occurrence rate ϵ_i in the training set. 
         
        Local AU language generation auxiliary training provides detailed semantic supervision for AU predictions and is optimised by commonly used negative log-likelihood loss, as follows:
        
    ℒ_Lgen = - 1/N∑_i=1^N1/T∑_t=1^T ( logp(s^l_t; i|V̂_i, s^l_[0:t-1]; i))
 
        where N is the number of AU branches and T is the length of each description. The t-th word s^l_t; i of AU_i description is generated based on the previous words s^l_[0:t-1];i and the AU_i refined visual feature V̂_̂î. 

        Moreover, the introduced global language generation is also optimized by a similar objective function in Eq. (<ref>) with the global generation S^g = [s^g_1,...,s^g_T]  as ℒ_Ggen:
        
    ℒ_Ggen = -  1/T∑_t=1^T ( logp(s^g_t|V, s^g_[0:t-1]))
 
        Different from the local language generation for specific AU branches, the global language generation faces the challenge of linguistic semantic diversity due to facial state changes. To this end, we add a global AU classification loss ℒ_Gau together with ℒ_Ggen as a constraint, where AU states are predicted by a shared linear classifier based on the average attention-aware visual feature from language model.

        Finally, the joint loss of our VL-FAU model can be optimized by maximizing the following lower bound:
        
    ℒ =  ℒ_Fau + ℒ_Lgen + ℒ_Ggen + ℒ_Gau.
         



§ EXPERIMENTS





    

 §.§ Dataset and Implementation Details

    
    Dataset.
        We provide evaluations on the widely used datasets, i.e. BP4D <cit.> and DISFA <cit.>.
        BP4D is a spontaneous FAU database containing 328 facial videos from 41 subjects (23 females and 18 males) involved in 8 sessions. Similar to <cit.>, the most expressive frames are manually labeled with AU occurrence from each session (around 400-500 frames). Overall, BP4D contains 12 AUs and 140K valid frames with labels. 
        
        DISFA consists of 27 participants (12 females and 15 males). Each participant has a video of 4,845 frames. 8 AU annotations are selected following the popular studies <cit.>. 
        Compared to BP4D, the experimental protocol and lighting conditions deliver DISFA to be a more challenging dataset. 
        
    Data Processing[The details of language descriptions are in https://github.com/XuriGe1995/VL-FAU.].
        During training, language descriptions of different AU states are hand-crafted, containing descriptions of both activated and inactivated AU states. Each local AU description contains multiple muscle details with potential associations according to FACS <cit.>. The global face description is generated from the AU annotations and FACS <cit.>, which only focus on the activated AU muscles. The examples are shown in Figure <ref>.
        We evaluated the model using the common 3-fold subject-exclusive cross-validation protocol <cit.>.



    Training strategy.
        The whole end-to-end network is implemented with PyTorch on a single NVIDIA RTX 3090Ti GPU using AdamW solver with β_1 = 0.9, β_2 = 0.999, and a weight decay of 0.0005. Maximum epochs are set to 15 with a batch size of 64.
        During training process, aligned faces are randomly cropped into 224 × 224 and horizontally flipped. We randomly employ the cutout augmentation <cit.> to overcome overfitting and improve the robustness during training. The stem extractor (Swin-Transformer-base) is pre-trained on ImageNet. The dimensionality of the global feature is mapped from 1024 to 512 (d=512), matching the hidden state dimension (h=512).

        

 §.§ State-of-the-art Comparisons

    We perform extensive experiments to compare our VL-FAU with mainstream FAU recognition studies and the latest state-of-the-art methods on two widely used FAU benchmarks in Table <ref> and Table <ref>. 
    The best and second-best results are bold and underlined. 
        

    Evaluation metrics.
        For all methods, the frame-based F1 score (F1-frame, %) is reported, which is the harmonic mean of the Precision P and Recall R and calculated by F1=2PR/(P+R). To conduct a more comprehensive comparison with other methods, we also evaluate the accuracy performance (%). In addition, the average results over all AUs (denoted as Avg.) are computed with “%” omitted.


     

   
    
    Quantitative comparison on DISFA: 
     We compare our proposed VL-FAU with its counterpart in Table <ref> and Table <ref>. 
     Our VL-FAU outperforms mainstream studies with impressive margins.
     In particular, compared with the state-of-the-art AAR <cit.>, which joint surprised local attention maps to obtain the multi-branch attention-aware AU representation, our VL-FAU increases the average F1-frame by 2.3% and shows clear improvements for most annotated AU categories. 
     Compared with the best model KDSRL <cit.> on DISFA, the average F1-frame score of our VL-FAU is also improved from 64.5% to 66.5%. 
     Furthermore, we achieve the best performance in terms of average accuracy in Table <ref>, compared with all methods.
     
     


          
    Quantitative comparison on BP4D:  FAU recognition results by different methods on BP4D are shown in Table <ref> and Table <ref>, where the proposed VL-FAU model achieves a new state of the art compared with all methods in terms of average F1-frame score. 
    Our VL-FAU outperforms the baseline JAA-Net <cit.>, which integrates AU detection and face alignment, in terms of average F1-frame and accuracy by 5.8% and 1.4%, respectively. 
    This is mainly because JAA-Net focuses on the local AU regions based on the detected landmarks, resulting in poor distinguishability between different AUs, especially for the same individual with subtly different face states, which can be improved by our method. 
    Moreover, VL-FAU achieves the best or second-best F1 and accuracy scores in recognizing most of the 12 AUs in BP4D, outperforming other state-of-the-art methods.
    
   

     
    In addition, compared with SEV-Net <cit.>, which prior encodes the pre-provided linguistic descriptions into image features, our VL-FAU achieves 7.7% and 1.9% higher average F1-frame scores on DISFA and BP4D, respectively. 
    Experimental results demonstrate the effectiveness of VL-FAU in improving AU recognition accuracy on DISFA and BP4D by our proposed joint learning with language generation. 
    Besides, as shown in Figure <ref>, we also provide the multi-label performance balancing analysis on DISFA and BP4D. Although the performance varies greatly between different AUs due to the inherent category imbalance in datasets, our VL-FAU still achieves a better performance-balance than existing methods and maintains the best overall results. 
    

 

    

 §.§ Ablation Studies

    We perform extensive ablation studies on BP4D to investigate how each component affects the overall performance of the proposed VL-FAU. 
    Due to space limitations, we do not show the ablation results for DISFA, but it is consistent with BP4D.
    Table <ref> presents component ablation studies focusing on the various modules within VL-FAU, including (1) multi-level AU representation learning (MARL), (2) local language generation auxiliary (LLGA), and (3) global language generation auxiliary (GLGA). 
    In addition, Figure <ref> gives a qualitative analysis of local and global language generations.

    

    (1) Multi-level AU Representation Learning (MARL). 
    
    Compared with the baseline model 172, the result has an improvement of average F1-frame from 64.3% to 64.7% when considering the proposed multi-level AU representation learning (indicated variant 173 with MARL). 
    It indicates that the dual-level individual AU refinement based on the multi-scale stem feature combination could get richer and more fine-grained AU features and hence improve recognition performance. 


    
    (2) Local Language Generation Auxiliary (LLGA).
    In Table <ref>, we test the contributions of local language generation auxiliary (LLGA) of our VL-FAU model.  
    Compared with variant 173 and variant 174, after we provide local language generation auxiliary for each AU branch, the average F1-frame score has been improved from 64.7% to 65.3%.
    In addition, most of the 12 AUs annotated in BP4D achieve significant improvements.  
    
    These observations demonstrate that by providing each AU branch with local language generation as an explicit auxiliary semantic supervision, the discriminative ability between AUs becomes stronger due to the gain of language expressiveness rather than single visual appearance features.


  
    
    (3)  Global Language Generation Auxiliary (GLGA).
    Similarly, we conduct a comparison between variant 173 and variant 175 to verify the effectiveness of the proposed global language generation auxiliary (GLGA) for whole-face representation learning. 
    Results in Table <ref> show that the proposed VL-FAU facilitates the target FAU recognition task when using GLGA (indicated variant 175). 
    In particular, the averaged F1-frame score increases from 64.7 % to 65.2 % and most AUs achieve significant improvements.  
    These validate the effectiveness of the proposed GLGA which provides better discriminability of global representations by focusing on the language semantics of activated facial AUs, especially for the same subject with subtly different AU states.


    Finally, when both local and global language generation auxiliaries (indicated model 176) are considered, the proposed VL-FAU achieves the best recognition performance in terms of average F1, significantly better than the single variants (variant 174 and 175) and variant 173 in Table <ref>.  
    In addition, the local and global language generation performances also have certain improvements, achieving 86.6% and 64.7% on top-5 accuracy of word generation in LLGA and GLGA. 
    These quantitative comparisons experimentally demonstrate that exploring explicit semantic-auxiliary supervisions for facial AU recognition is a beneficial way for discriminating different AU states under intra- and inter-subject.


    
    (4) Qualitative Analysis of Local and Global Generation Auxiliary.
    Besides the above quantitative comparisons, we further proved the detailed qualitative analysis of our main innovations – local and global generation auxiliaries for facial AU recognition.
    As shown in Figure <ref>, we use t-SNE <cit.> to visualize the AU features learned by the proposed VL-FAU and the corresponding baseline without local and global generation auxiliaries (baseline 173). Note that, both models are used the same multi-branch networks with MARL. 
    Specifically, we extract the AU features of 8 random gender-balanced subjects for clearer visualization from multiple AU branches before the final classification and then visualize them by t-SNE. We provide different coloring schemes in Figure <ref> to analyze the impacts of VL-FAU on different aspects, including AU category, subject ID, and gender. 
    (1) Comparisons of clustered AU features in the first row, our VL-FAU can better divide different AU features into different clusters, while the baseline is not sufficiently discriminative on some AU features (marked with red circles).
    Besides, we notice that AU features optimized from baseline are mapped in a narrow space compared with our VL-FAU.
    These observations indicate that by joining language generation auxiliary, our VL-FAU can maintain higher discriminability between multiple AU representations. 
    (2) The second row shows the subject ID coloring results. Our VL-FAU distinguishes different subject AU features more widely within the same AU cluster, indicating that our VL-FAU provides higher discriminability between different subjects for facial AU recognition.
    (3) Due to the explicit language supervision of gender information in GLGA, VL-FAU can achieve clearer gender colorization results, as shown in the last row of Figure <ref>.
    
    
  


    
    

 §.§ Visualization of Results

     To further understand the quality of the proposed vision-language joint learning for FAU recognition and description generation, we visualize the predicted AU states and their corresponding local and global-level descriptions, as shown in Figure <ref>. 
     Two positive examples from BP4D contain visualizations of different genders with different AU states. Compared with the mainstream paradigm, our VL-FAU provides explainable FAU recognition with language generations. In detail, local descriptions contain multiple detailed muscle changes with natural connections, improving intra-AU semantics and inter-AU distinguishability. 
     In addition, the global descriptions contain diverse activated AU states with gender information, which can improve the inter-face distinguishability within and between subjects. 
     Besides, we provide a bad case, which makes wrong predictions in two AUs. However, the global description ignores the misrecognition. Although AU6 is incorrectly predicted, the detailed description matches the facial expression, possibly due to labeling ambiguity. 
     Overall, our VL-FAU can give better explainable facial AU recognition with explicit local and global language descriptions.
     
    


§ CONCLUSION AND FUTURE WORK

    In this paper, we proposed a novel end-to-end vision-language joint learning (VL-FAU) for explainable FAU recognition along with language generations as explanations. 
    As auxiliary supervisions, local and global language generations are joined into a multi-branch AU recognition network with multi-level AU representation learning. 
    Local AU language generation provides explicit fine-grained semantic supervision for each AU classification with detailed language descriptions, improving the discrimination of inter-AU representations. 
    Global language generation, employing multi-scale combined stem features, offers diverse semantic supervision for the whole facial feature to maintain diversity and distinction across intra- and inter-subject representation changes. 
    Our VL-FAU finally provides predictions of AU states as well as interpretable language descriptions for individual AUs and global faces.
    Extensive experimental evaluations on DISFA and BP4D show that our VL-FAU outperforms state-of-the-art AU recognition methods with impressive margins. 
    
    VL-FAU introduces a traditional language model for explainable FAU recognition considering computational power and efficiency limitations, saving about 200% in inference time and GPU memory usage. We believe our attempts provide new inspiration for multimodal multi-task joint training for explainable FAU recognition. In the future, we would like to investigate the further combination of FAU recognition with popular LLMs for more diverse and fine-grained explainable generations.  







This work was supported in part by the MUR PNRR project FAIR (PE00000013) funded by the NextGenerationEU and by the PRIN project CREATIVE (Prot. 2020ZSL9F9). Fuhai Chen's research was supported by the Fujian Provincial Department of Education Youth Project, China (grant JZ230006) and the Engineering Research Center of Big Data Intelligence, Ministry of Education, China, and Fujian Key Laboratory of Network Computing and Intelligent Information Processing (Fuzhou University), China.





ACM-Reference-Format





 Paper Title: RDP: Ranked Differential Privacy for Facial Feature Protection in Multiscale Sparsified Subspace.








RDP: Ranked Differential Privacy for Facial Feature Protection in Multiscale Sparsified Subspace

    Lu Ou
Hunan University

Changsha, China 

oulu9676@gmail.com
Shaolin Liao
IIT

Chicago, USA 

sliao5@iit.edu
Shihui Gao
Hunan University

Changsha, China 

gaoqaq@hnu.edu.cn
Guandong Huang
State Grid Hunan Materials Company

Changsha, China 

huanggd@hnu.edu.cn
Zheng Qin
Hunan University

Changsha, Country 

zqin@hnu.edu.cn







    
================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
With the widespread sharing of personal face images in applications' public databases, face recognition systems faces real threat of being breached by potential adversaries who are able to access users' face images and use them to intrude the face recognition systems. Although large amounts of work on facial features privacy preserving methods have been proposed, it is difficult to provide accurate and lightweight protection without degrading the visualization quality of face images, or data utility, under the constraint of given privacy budget for effective facial features protection. In this paper, we propose a novel privacy protection method in the multiscale sparsified feature subspaces to protect sensitive facial features, by taking care of the influence or weight ranked feature coefficients on the privacy budget, named “Ranked Differential Privacy (RDP)”. After the multiscale feature decomposition, the lightweight Laplacian noise is added to the dimension-reduced sparsified feature coefficients according to the geometric superposition method. Then, we  rigorously prove that the RDP satisfies ε_0-Differential Privacy. After that, the nonlinear Lagrange Multiplier (LM) method is formulated for the constraint optimization problem of maximizing the utility of the visualization quality protected face images with sanitizing noise, under a given facial features privacy budget ε_0. Then, two methods are proposed to solve the nonlinear LM problem and obtain the optimal noise scale parameters: 1) the analytical Normalization Approximation (NA) method where all Laplacian noise scale parameters are normalized to the average noise scale parameter, which is good for real-time online applications; and 2) the LM optimization Gradient Descent (LMGD) numerical method to obtain the nonlinear solution through iterative updating, which is better for more accurate offline applications. Experimental results on two real-world datasets show that our proposed RDP outperforms other state-of-the-art methods: at a privacy budget of ε_0 = 0.2, the PSNR (Peak Signal-to-Noise Ratio) of the RDP is about ∼ 10 dB higher than (10 times as high as) the highest PSNR of all compared methods.



data privacy, multiscale feature space, differential privacy, optimization, face image databases.




§ INTRODUCTION

In the current digital age, it has become a common phenomenon that individuals' static face images are shared in the public domain, such as Facebook, TikTok, Bing, <cit.>. Moreover, there are universal and unique facial features with rich biometric information in such images. Existing research shows that facial features provide a solid foundation for face recognition techniques, which plays a vital role in applications such as identity authentication and access control. For example, the face recognition payment function launched by Alipay allows a user to verify his/her identity through scanning his/her face <cit.>; and it is a common identity authentication application to filter permissions using face recognition techniques <cit.>. Unfortunately, adversaries could breach face recognition systems by public face images <cit.>. Then they can use smart devices without authorization <cit.>, access private systems <cit.>, and even obtain personal sensitive information <cit.>, . As shown in Figure <ref>, due to the sensitive facial features in face images, once adversaries have obtained original face images in public databases, they could breach face recognition-based systems. From that, it means that facial features in public face images not only threaten personal privacy, but also may lead to a series of security threats, including identity theft, fraud, and even political and economic espionage, which has been brought up by <cit.>. What's more, the privacy leakage even could be caused by only one face image <cit.>. All of that said, it is necessary to develop an effective facial feature protection method for public face images.


In existing research, cryptography  <cit.>, deep learning,   <cit.> and Differential Privacy (DP) <cit.> are adopted to protect facial features effectively. Although cryptography-based facial feature privacy protection methods can provide strong security, they usually focus on defending against untrusted models or servers, and have expensive computation and communication overhead. Besides, deep learning<cit.> needs to be trained on large-scale data sets to avoid privacy risks, which is more used for preserving face recognition accuracy while protecting facial features. Different from the previous two kinds of methods, DP <cit.> is a solid mathematical method that can adds sanitizing noise for protecting facial features while preserving not only the recognition accuracy but also the visual quality of images with released demand. What's more important is that DP can be optimized to maximize the data utility of face images after the sanitizing noise is added, under the constraint of a given privacy budget. Existing methods based on DP <cit.> may be more likely to add perturbation during the training process, which comes with a large training cost.



In this paper, the data utility is the visualization quality of the noisy face images after the sanitizing noise is added. And the sensitive information is the facial features in face images that could be used to breach some face recognition-based systems. For hiding sensitive information of each individual's face image and conserving the face image's data utility as far as possible, the perturbation on the facial features should be minimized while maximizing each noisy face image's visualization quality with a given privacy budget of this image's facial features. 

And then, we propose such an optimized facial features protection method based on Ranked Differential Privacy (RDP): first, we perform the dimension-reduced sparsification of each original face image for extracting the original facial details into the multiscale feature space and obtaning sparse feature coefficients; then, the influence or weight of each sparse feature coefficient on the total privacy budget ε_0 is evaluated; finally, sanitizing noise of the optimal scale parameters is added to the ranked significant sparse feature coefficients by taking care of the influence or weight of each coefficient. The main contributions of this paper are as follows,


  * First, after reducing the dimension of every face image and obtaining the sparse feature coefficients, the weights of each coefficient with respect to the facial feature vector and thus the total privacy budget ε_0 are evaluated and coefficients are ranked in accordance with such weights. Then, Lagrange Multiplier method (LM) constraint optimization problem is formulated: optimizing the Laplace scale parameters of all feature coefficients with significant influence or weights to  maximize the visualization quality of the privacy-preserving noisy face image, under the constraint of the total privacy budget ε_0 of the face image's facial features. 

  * What's more important, two methods are proposed to solve the nonlinear LM optimization problem: the Normalized Approximation (NA) analytical method by approximating all weighted Laplacian noise scale parameters to be identical, which is suitable for real-time online applications; and the iteratively updating LM optimization Gradient Descent (LMGD) numerical method, which is first proposed here to achieve more accurate offline applications.

  * Finally, experiments on two real-world datasets show that the facial features have been effectively protected, while the face image's visualization quality is only affected minimally, compared to other state-of-the-art DP methods.






§ RELATED WORK
































Due to facial features that existed in face images, the images could be used to breach face recognition-based systems <cit.>. There is the related research on facial features protections. Then, we will review the research in recent years. 

A lot of research focus on the problem of how to reserve the efficiency of face recognition while preventing facial features in face images from being leaking for individuals' sensitivity information or face images reconstruction: Chamikara   <cit.> proposed a face recognition privacy protection scheme based on Local Differential Privacy (LDP), which calculates by perturbing the eigenface and storing the perturbed data in a third-party server;  
Ding <cit.> proposed a model with dual channels and multiple monitors to prevent DeepFake anti-forensic attacks for protecting facial features; Meng   <cit.> proposed a face privacy framework, which extracts purified clusters locally through Differentially Private Local Clustering (DPLC) method and then encourages global consensus among clients to generate more discriminative features; Wang <cit.> considered that facial features used by face recognition system for authentication may be attempted to reconstruct the original face image and proposed a privacy protection method based on disturbing the mapping from facial features to the image; Lin <cit.> proposed a privacy-preserving facial features based on a new facial recognition paradigm with a feature compensation mechanism for the challenge between the public demand for powerful face recognition and the individuals' privacy concerns; Mi <cit.> proposed the MinusFace facial features privacy protection framework for enhancing facial feature protection by random channel shuffling and achieving high accuracy in face recognition system. Others are more interested in preventing adversaries from attacking face recognition models or servers storing the relevant data to obtain facial features: Wang and Leng <cit.> proposed an image encryption algorithm that combines face recognition and bitonal sequence technologies, and designed a row-column scrambling algorithm to scramble facial features through the characteristics of bitonal sequences; Zhuo <cit.> proposed an AdaBoost-based framework POR to protect the privacy of users' facial features and the service provider's key learning parameters in face recognition integrated deep learning algorithms. Additionally, there are some studies worrying about the facial features leaking of face images released to the public domain: Ji <cit.> proposed a frequency-domain DP protection method DCT-DP for facial features, with an optimized privacy budget allocation method based on the loss of the face recognition network; Croft <cit.> proposed a face obfuscation DP method for generating machine learning models, adding noise to the image pixel intensities for protecting facial features.

All above, such research effectively protects individuals' facial features in face images. However, most studies are more concerned with the trade-off between face recognition accuracy and faial features protection, or consider the unreliability of face recognition models and servers storing related data. In this paper we intend to solve how to release face images with high-quality visual effects on the premise of protecting facial features. Different from <cit.>, we attempt to propose a lightweight perturbation method with adding less noises than <cit.> in the face images not in training model.







§ PRELIMINARY WORK

In this section, we will formally define the notations involved in this paper and systematically introduce models and problem statement. 


At beginning, notations are shown in Table <ref>.







 §.§ Models and Problem Statement

Next, we will give the system and adversary models, and state the problem. 



  §.§.§ System Model
 
With a certain individual 𝔸, there exist a large amount of 𝔸's static face images represented as a dataset Ω in public, which can be unrestricted observed and obtained. F̅^( A̅) is the standard feature vector retrieved from 𝔸's standard face A̅, used for authentication in some systems with face recognition techniques. For any face image A visually similar to the individual 𝔸, its feature vector is F̅ in the same face recognition with F̅^( A̅). The system identifies that individual 𝔸 is passing and can pass through A, if F̅ and F̅^( A̅) meet the following matching criterion

    | F̅_̅i̅ - F̅_̅i̅^( A̅)| ≤R̅_i;  i = 1,2, ⋯ ,M_F,

where F̅_̅i̅ and F̅_̅i̅^( A̅)  are the i-th element of F̅ and F̅^( A̅) with M_F elements respectively, and R̅_i is the i-th element of the feature identification radius vector R̅ for this authentication system.











  §.§.§ Adversary and Problem






























In this paper, adversaries are considered to be interest to and have the ability to download the face image dataset Ω of the certain individual 𝔸 from the public domain, and they could obtain information about authentication devices equipped with the face recognition system, which are commonly used by individual 𝔸. Adversaries may attempt to breach these face recognition system by using such face image in Ω, and pass the authentication to obtain 𝔸's privacy information.

For the face image dataset Ω of 𝔸, our goal is to develop an accurate and lightweight facial feature vector privacy-preserving method f' that sanitizes these images by adding noise, A' = f'( A ) for any face image A  ∈Ω. Such method can ensure that the noisy face image A' can prevent adversaries from breaching face recognition-based systems while having extremely high visual quality, meaning that we aim to optimize the utility of the sanitized images by maximizing images' visualization quality while providing privacy protection for facial features in face images.












 §.§ Basic Concepts

In this section, we give some preliminary definitions for DP. 



First, in the terminology of DP, we define a pair of neighboring face images, expressed in Definition <ref>.















[Neighboring Face Images]
If a face image A belongs to the individual 𝔸 and a face image B does not belong to the individual 𝔸, The face images A and B are a pair of neighboring face images.



Then, the sensitivity of facial features is defined as Definition <ref>.

[The Sensitivity of Facial Features]
If ℳ is a measurement of face images' facial features, its sensitivity is 

    Δ_i = max_A,B||ℳ(A)-ℳ(B)||_1,

where A and B are neighboring face images. 




































Finally, the definition of DP is as follows.

[Differential Privacy]
There is a measurement ℳ' and a pair of neighboring face images A and B. ℳ' is ε_0-differentially private, if the following is satisfied



    |[  ℳ'(A)=F̅^']/[  ℳ'(B)=F̅^']|≤exp( ε _0).































§ RANKED DIFFERENTIAL PRIVACY

In this section, we introduce our proposed Ranked Differential Privacy (RDP) for facial features with weighted sparsified subspace. Overall, before releasing the original face images, the Laplacian noise is added to the face images' facial features by geometric superposition <cit.>, in order to hide the facial features which are used in the process of face recognition of the systems. 























 §.§ Perturbation in Multiscale Feature Space














As well known, the face images' facial features play a key role in the implementation of face recognition. So that they should be protected to avoid adversaries' breaching face recognition-based systems. From this, before the perturbation, facial features should be extracted in the multiscale feature space. Because of its efficiency and rigorous mathematical expression, Multi-level Haar Wavelet Transform (HWT) <cit.> is adopted to reduce the dimensionality of the face image's pixels and obtain accurate information about the face image's facial features, , the wavelet coefficient vector constructed by the sparsified wavelet coefficients is obtained. After the perturbation on the wavelet coefficient vectors, the noisy face image could be generated by the noisy pixels matrix P' obtained by the noisy coefficients vector C̅' through Multi-level Inverse Wavelet Transform (MIWT).

During the extraction and dimensionality reduction of facial features from face images, we use the first-order Taylor expansion to describe the liner mapping relationship between the noise of the sparse wavelet space and the noisy reduced dimensionality facial features of the noisy face image. After adding Laplacian noise to the elements of the original wavelet coefficient vectors C̅, we obtain the noisy feature vector F̅' by the noisy face image, which comes from the noisy wavelet coefficient vectors C̅' with MIWT.  According to the first-order Taylor formula for the i-th element F̅_i^' of the noisy facial feature vector F̅', we have

    F̅_i^'    = F̅_i + ∑_j = 1^M_P∂F̅_i^'/∂C̅_j^'·( C̅_j^'  - C̅_j)  + o( C̅'),

where 1 ≤ i ≤M_F; C̅_j^' is the j-th element of C̅', where M_P is the number of the elements of the face image's pixel matrix and 1 ≤ j ≤ M_P; and o( C̅') is the minimal term, , o( C̅') ∼ 0. Thus, we think

    F̅_i^'  = F̅_i + ∑_j = 1^M_P∂F̅_i^'/∂C̅_j^'·( C̅_j^'  - C̅_j)  .

Obviously, C̅_j^'  - C̅_j represents the noise of C̅_j^', and the noise of F̅_i^' can be considered as the weighted sum of elements' noises of C̅', with that the first-order partial derivative of F̅_i^' on C̅_j^' is regarded as the weight for the noise of C̅_j^'. 

Then, we define the weight w_ik for the i-th element F̅_i^' of the noisy feature vector F̅' and the k-th element C̅_k^ * ^' of the sorted noisy wavelet coefficient vector C̅^ * ^' as w_ik = ∂F̅_i^'/∂C̅_k^ * ^'. 

After this, with the perturbation provides by weighted sum, we can select the wavelet coefficients more significant in facial features from the sorted wavelet coefficients basis of such weights for adding Laplacian noises with appropriate scale to wavelet coefficients, thereby providing stricter privacy preservation. The definition of the noise mechanism based on such ranked wavelet coefficients is shown in Definition <ref>.

[Ranked Differential Privacy]
Given the geometric number K between 1 and M_P with mean 1/p, 0 < p < 1. For obtaining  the noisy wavelet coefficients vector C̅', K Laplacian noises with mean 0 are added to the top K of the wavelet coefficients vector C̅'s ranked elements of a face image, with ranking elements of C̅' by the corresponding weights between the noisy feature vector's elements and such elements. After the inverse transformation with MIWT of C̅', the noisy pixel matrix P' is obtained from C̅', and the noisy face image is generated by P'. Such method,  providing perturbation to the elements of the feature vector F̅ by the weighted sum of ranked wavelet coefficients' Laplacian noises for protecting facial features, is called Ranked Differential Privacy, referred as "RDP".



















In RDP, after perturbation of wavelet coefficients, the noisy pixels matrix P' is obtained by C̅^ * ^' through MIWT, and the noisy feature vector of the noisy face image generated by the noisy face image from P' is F̅' with M_F elements. Let the noise of the i-th element F̅_i^' in F̅' be ζ _i, F̅_i^'  = F̅_i + ζ _i, with  1 ≤ i ≤M_F. Then we have Theorem <ref> of RDP about F̅'.


The noise ζ _i of the i-th element F̅_i^' in F̅',which is  obtained by RDP, satisfies Laplace distribution, , [ ζ _i] = Lap( 0,Δ _i/ε _i), where ε _i is the privacy budget for the i-th element F̅_i^' of F̅', and 1 ≤ i ≤M_F.






Let C̅^*^' and C̅^*  be the noisy ranked wavelet coefficient vector and the ranked wavelet coefficient vector, separately. According to Eq. (<ref>), we have the following first-order Taylor expansion for F̅_i^',

    F̅_i^' = F̅_i + ∑_k = 1^M_P∂F̅_i^'/∂C̅_k^ * ^'·( C̅_k^ * ^' - C̅_k^ * )  + o( C̅^ * ^'),

where C̅_k^*^' and C̅_k^* are the k-th element of C̅^*^' and C̅^*, separately, and o(C̅^ *^') are the minimal term, , o( C̅^*^') ∼ 0. Thus, we think

    F̅_i^'  = F̅_i + ∑_k = 1^M_P∂F̅_i^'/∂C̅_k^ * ^'·( C̅_k^ * ^' - C̅_k^ * ) .


The weight for the i-th element F̅_i^' of the noisy feature vector F̅' and the k-th element C̅_k^ * ^' of the sorted noisy wavelet coefficient vector C̅^ * ^' is w_ik, w_ik = ∂F̅_i^'/∂C̅_k^ * ^'. K is the given geometric number between 1 and M_P with mean 1/p, 0 < p < 1. After we add Laplacian noises to the top K elements of the ranked wavelet coefficient vector C̅^ * according to RDP, the first-order Taylor expansion of F̅_i^' is further

    F̅_i^'  = F̅_i + ∑_k = 1^M_P∂F_i^'/∂C̅_k^ * ^'( C̅_k^ * ^' - C̅_k^ * )  = F̅_i + ∑_k = 1^M_Pw_ikξ _k  .

Then due to there are K noises, ξ _k = {[ ξ _k,   1 ≤ k ≤ K;   0,   k ≤ k ≤M_P ]., and F̅_i^'  = F̅_i + ∑_k = 1^K w_ikξ _k. From that, we have ζ _i = ∑_k = 1^K w_ikξ _k, and can get the probability of ζ _i, [ ζ _i] = [ ∑_k = 1^K w_ikξ _k]. And based on the geometric superposition, ζ _i satisfies Laplace distribution, , ζ _i∼ Lap( 0,Δ _i/ε _i) with sensitivity Δ _i and privacy budget ε _i of F̅_i^'. Especially, the scale parameter b_k of the noise ξ _k is related to the privacy budget ε _i of F̅_i^', 1 ≤ k ≤ M_P.

This theorem has been proved.




Next, the distribution of the noisy feature vector F̅' can be derived from elements of F̅'. Based on Theorem <ref>, we can obtain the relationship between the total privacy budget of F̅' and the scale parameters of Laplacian noises in C̅^ * ^'.




 §.§ Privacy Budget Analysis

In face recognition, the facial feature vector is obtained by projecting the face image onto the eigenfaces, which are constructed by a set of orthogonal eigenvectors through SVD decomposition and dimensionality reduction from the original face images dataset <cit.>. Such eigenvectors capture the most significant variations of features and represent the main change directions in the feature space, and their corresponding singular values represent the importance of eigenfaces. Due to their orthogonality, these eigenvectors are independent and can serve as the basis vectors of the feature space, enabling the representation of the original face image. Consequently, the feature vector's elements, which are projections of the original image onto these eigenvectors, exhibit independence. According to Theorem <ref>, ζ _i = ∑_k = 1^K w_ikξ _k, where K is the given geometric number with mean 1/p, 0 < p < 1, and w_ik is the weight between F̅_i^' and C̅_k^ * ^', w_ik = ∂F̅_i^'/∂C̅_k^ * ^', 1 ≤ k ≤M_P. Let the sensitivity Δ _i and the privacy budget ε _i of F̅_i^', we have [ ζ _i] = Lap( 0,Δ _i/ε _i). According to the variance operation, the variance of ζ _i is 

    Var[ ζ _i] = 2( Δ _i/ε _i)^2.

and the variance of ∑_k = 1^K w_ikξ _k is

    Var[ ∑_k = 1^K w_ikξ _k]    = ∑_K = 1^M_Pp( 1 - p)^K∑_k = 1^K Var[ w_ikξ _k]
        = 2∑_K = 1^M_Pp( 1 - p)^K∑_k = 1^K ( w_ikb_k)^2
        = 2∑_k = 1^M_P( w_ikb_k)^2∑_K ≥ k^M_Pp( 1 - p)^K
        = 2∑_k = 1^M_P( w_ikb_k)^2[ ( 1 - p)^k - ( 1 - p)^M_P + 1].


Because of Var[ ζ _i] = Var[ ∑_k = 1^K w_ikξ _k] from [ ζ _i] = [ ∑_k = 1^K w_ikξ _k], we get the expression of ε _i with respect to b_k as 

    ε _i( b_k) = Δ _i/√(∑_k = 1^M_P( w_ikb_k)^2[ ( 1 - p)^k - ( 1 - p)^M_P + 1]).

And based on that the elements of F̅' are independent, the total privacy budget of F̅' is ∑_i = 1^M_Fε _i( b_k), then we have the total privacy budget's expression ε( b_k) of F̅' with respect to b_k as

    ε( b_k) = ∑_i = 1^M_FΔ _i/√(∑_k = 1^M_P( w_ikb_k)^2[ ( 1 - p)^k - ( 1 - p)^1+M_P]).


















 §.§ Optimal Data Utility under Privacy Budget


Our optimization goal is to maximize the noisy face image's visualization quality under the constraint of the privacy budget, , optimizing the noise scale parameters of all wavelet coefficients, which is a constraint optimization problem.

Since the variance of the noise quantifies the negative impact of the noise on the visualization quality and a constant does not have influence on the variance, the cost function for the visualization quality of the face image can be defined by deriving the variance of the noisy pixel matrix, of which value is smaller with the smaller loss of visualization quality. 

Further, the constrained optimization problem can be converted into minimizing the cost function under the constraint of privacy budget. Then, we define the cost function as Definition <ref>.

[Cost Function]






After multi-level HWT for the original face image, C̅ is the one-dimensional vector that the wavelet coefficient matrix expansion to. C̅^ * is a one-dimensional vector after sorting the elements of C̅. According to RDP (Definition <ref>), the Laplacian noises are added  to C̅^ * and get the ranked noisy one-dimensional wavelet coefficient vector C̅^ * ^'. Define that b_k is the scale parameter of the Laplacian noise of the k-th element of  C̅^ * ^', 1 ≤ k ≤M_P, then the expression of the cost function f( b_k) of the noisy face image with respect to b_k is 

    f( b_k) = ∑_k = 1^M_P( b_k)^2[ ( 1 - p)^k - ( 1 - p)^1+M_P].













Given the pixel matrix P of a face image, and it undergoes a multi-level wavelet transform to obtain a matrix of wavelet coefficients C. Assume that the transformation matrix of wavelet transform is H̃ <cit.>, which is a √(M_P)×√(M_P) orthogonal matrix. After N-level wavelet transform, let H = H̃^N, and H^T = ( H̃^T)^N. Then 

    C = H̃^NP = HP, and P = ( H^T)^NC = H^TC,

where H^TH = I, and I is an identity matrix.

Define the Laplacian noise matrix of wavelet coefficient matrix C is N, and the noisy wavelet coefficient matrix is C' = C + N. Then for the noisy pixel matrix P',

    P' = ( H̃^T)^NC' = H^T( C + N) = P + H^TN.










Let B be the Laplacian scale parameter matrix of N, N_mn∼ Lap( 0,B_mn), 1 ≤ m ≤M_P, 1 ≤ n ≤M_P. Thus,

    [                                                              Var[ P'] = 2∑_m = 1^√(M_P)∑_n = 1^√(M_P)∑_t = 1^√(M_P)[ ( H^T)_mt]^2·( B_tn)^2;                                                                           = 2∑_n = 1^√(M_P)∑_t = 1^√(M_P)( B_tn)^2∑_m = 1^M_P[ ( H^T)_mt]^2;                                                                                                    = 2∑_n = 1^√(M_P)∑_t = 1^√(M_P)( B_tn)^2;                                                                                                                                = Var[ C'] . ]


For the convenience of calculation, we flatten the pixel matrix P and the wavelet coefficient matrix C into the one-dimensional pixel vector P̅ and the wavelet coefficient vector C̅. C̅^ * is a one-dimensional vector after sorting the elements of C̅. Then the variance of the noisy pixel vector P̅' and the noisy wavelet coefficient vector C̅' is equal, 
    [                                                Var[ P̅']                                         = Var[ C̅^ * ^'];                                                                       = 2∑_K = 1^M_Pp( 1 - p)^K∑_k = 1^K ( b_k)^2;                                                          = 2∑_k = 1^M_P( b_k)^2[ ( 1 - p)^k - ( 1 - p)^M_P + 1] , ]

where C̅^ * ^' is the ranked noisy one-dimensional wavelet coefficient vector.

From that, the expression of the cost function of P̅' with respect to b_k is as follows 

    f( b_k) = ∑_k = 1^M_P( b_k)^2[ ( 1 - p)^k - ( 1 - p)^M_P + 1]   ,

which is the cost function of the noisy face image.







From that, a constant ε_0 is given as the total privacy budget of the noisy feature vector F̅', and the constraint is ε( b_k) = ε_0. Thus, the optimization problem is as follows,

    b_k = argmin_b_k{f( b_k)|ε( b_k) = ε _0}.


We solve this optimization problem according to the Lagrangian Multiplier method (LM) to better balance the need to sanitize the facial features and preserve the visualization quality of the face image. According to LM, the Lagrangian function with respect to b _k is

    L( b_k,λ) = f( b_k) + λ( ε( b_k) - ε _0),

which can be solved by setting the first derivatives with respect to λ and b_k to zeros,

    {[   ∂ L( b_k)/∂λ = 0;; ∂ L( b_k)/∂b_k = 0, ].




where the first equation reduces to the following,

    ∑_i = 1^M_FλΔ_i( w_ik)^2/{∑_k = 1^M_P( w_ikb_k)^2[ (1-p)^k-(1-p )^1+M_P]}^3/2 = 2.


It is clear that Eq. (<ref>) is nonlinear. Here we propose two ways to solve the nonlinear LM  equations for two different applications, , 1) the Normalization Approximation (NA) for the real-time online applications; and 2) the LM optimization Gradient Descent (LMGD) for the offline applications that need more accurate result.





  §.§.§ Optimization with Normalization Approximation

It is clear that Eq. (<ref>) is nonlinear,  which makes it difficult to find the analytical solution. Therefore, we propose the optimization with Normalization Approximation (NA), which assumes that all wavelet coefficients of interest contribute equally to the total privacy budget ε_0. 

To achieve such goal, it would be intuitive to choose the weighted Laplacian noise scale parameters b_k of Eq. (<ref>) to be the average value, , w_ikb_k≈b̅ = ∑_i = 1^M_Fw_ikb_k/M_F, where 1 ≤ i ≤M_F and 1 ≤ k ≤M_P. And substituting b̅ into Eq. (<ref>), we get

    ∑_i = 1^M_FλΔ _i( w_ik)^2/b̅^3{∑_k = 1^M_P( w_ik)^2[ ( 1 - p)^k - ( 1 - p)^1 + M_P]}^3/2  = 2.

It can be substituted by w_ikb_k to replace b̅ for obtaining the equations about every b_k and λ, which can be used to solve ∂ L( b_k)/∂b_k = 0. And then, Eq. (<ref>) can be solved as follows,

    b_k = ∑_i = 1^M_FΔ_i/w_ik/ε _0[1 - p - ( 1 - p)^1+M_P/p - M_P( 1 - p)^1+M_P]^1/2.




  §.§.§ LM Optimization via Gradient Descent

Although NA can solve the system of nonlinear equations in this optimization problem in real time, the assumption as its basis is are very idealistic, which may cause errors far from the best result. For the case allowing solving this problem offline, we propose solving the LM optimization problem via Gradient Descent (LMGD), which continuously tunes the initial value of b_k until an optimal solution is obtained through iterative gradient descent updating. 

It aims to find the best solution for b_k and λ by solving following equations,

    {[   ∂ L( b_k)/∂λ = 0;; ∂ L( b_k)/∂b_k = 0, ].
 which can be solved with two alternating gradient descent updates: 

1) gradient descent update with respect to b_k; 

2) gradient descent update with respect to λ. 

However, simple alternating gradient descent updates have the problem of finding the optimal step length or learning rate, which is difficult for non-convex optimization problems. Also, the nonlinear operation is required to achieve the minimum b_k, which motivates us to develop the LMGD to solve the nonlinear LM problem.  

According to LM, the purpose of LMGD is to obtain the optimal solution of b_k and λ,  through solving the following equations, 

    {[           ∑_i = 1^M_FΔ _i/√(∑_k = 1^M_P( w_ikb_k)^2[ ( 1 - p)^k - ( 1 - p)^1 + M_P])  - ε _0 = 0,; ∑_i = 1^M_FλΔ _i( w_ik)^2/{∑_k = 1^M_P( w_ikb_k)^2[ ( 1 - p)^k - ( 1 - p)^1 + M_P]}^3/2  - 2 = 0. ].

We define the error functions of two updating sub-tasks in LMGD as follows, 

    F_1( b_k^( h ))
    
     =    | ∑_i = 1^M_FΔ _i/√(∑_k = 1^M_P( w_ikb_k^( h ))^2[ ( 1 - p)^k - ( 1 - p)^1 + M_P])  - ε _0|;
      
       F_2( b_k^( h ),λ ^( h ))
    
     =    | ∑_i = 1^M_Fλ ^( h )Δ _i( w_ik)^2/{∑_k = 1^M_P( w_ikb_k^( h ))^2[ ( 1 - p)^k - ( 1 - p)^1 + M_P]}^3/2  - 2|,

where h represents the h-th parameter update. From that, the total loss function is

    F( b_k^( h ),λ ^( h )) = F_1( b_k^( h )) + F_2( b_k^( h ),λ ^( h )).


When F_1( b_k^(h)) → 0 with F_2( b_k^( h ),λ^( h )) → 0, the closer b_k and λ are to the optimal solution. 

The LMGD takes an initial solution b_k^( 0 ) of b_k and λ ^( 0 ) of λ as input, where ε( b_k^( 0 )) ≈ε _0, however ε( b_k^( 0 )) ε _0. After the initial values are inputted, the optimal solution of b_k and λ is obtained through the iterative updating process updated by two alternating gradient descent until the loss function F( b_k^( h ),λ ^( h )) is less than a minimum value δ, which is very close to 0. 

Given the step length or learning rate of η. Now the iterative updating process takes two alternating updates as follows,

(1) Step 1: firstly, b_k^(h) is updated iteratively by minimizing the loss function F_1( b_k^( h )) using the gradient descent, while b_k^(h) and λ ^(h) are updated iteratively by minimizing the loss function F_2( b_k^(h),λ ^( h )) using gradient descent, ,

    [                                                                                                   b_k^( h ) = b_k^( h ) - η[ ∂F_1( b_k^( h ))/∂ b_k^( h ) + ∂F_2( b_k^( h ),λ ^( h ))/∂ b_k^( h )];;                                                                                                                                         λ ^( h ) = λ ^( h ) - η∂F_2( b_k^( h ),λ ^( h ))/∂λ ^( h ), ]

where the gradient ∂F_1( b_k^( h ))/∂ b_k^( h ) can be obtained as follows

    [ ∂F_1( b_k^( h ))/∂ b_k^( h ) = | ∑_i = 1^M_FΔ _ib_k^( h )( w_ik)^2[ ( 1 - p)^k - ( 1 - p)^1 + M_P]/{∑_k = 1^M_P( w_ikb_k^( h ))^2[ ( 1 - p)^k - ( 1 - p)^1 + M_P]}^3/2|. ]


(2) Step 2: then, the gradients of ∂F_2( b_k^( h ), λ ^( h ))/∂ b_k^( h ) and ∂F_2( b_k^( h ),λ ^( h ))/∂λ ^( h ) are calculated by

    ∂F_2( b_k^( h ),λ ^( h ))/∂ b_k^( h )
    
     =    | 3∑_i = 1^M_Fλ ^( h )b_k^( h )Δ _i( w_ik)^4[ ( 1 - p)^k - ( 1 - p)^1 + M_P]/{∑_k = 1^M_P( w_ikb_k^( h ))^2[ ( 1 - p)^k - ( 1 - p)^1 + M_P]}^5/2|;
       ∂F_2( b_k^( h ),λ ^( h ))/∂λ ^( h )
    
     =    | ∑_i = 1^M_FΔ _i( w_ik)^2/{∑_k = 1^M_P( w_ikb_k^( h ))^2[ ( 1 - p)^k - ( 1 - p)^1 + M_P]}^3/2|.


(3) Step 3: finally, after alternatively repeating Step 1 and Step 2 above, the optimal solution of b_k and λ are obtained until F( b_k^( h ),λ ^( h )) < δ. 

The process by Eq. (<ref>) is proved as follow:


If the noise's scale parameter of the i-th element F_i^' of the noisy feature vector F̅' is b_i^( F̅), then the expression of b_i^( F̅) with respect to b^( h )_k is

    b_i^( F̅)( b_k^( h )) = √(∑_k = 1^M_P( w_ik b_k^( h ))^2[ ( 1 - p)^k - ( 1 - p)^M_P + 1])  ,

and the expression of the total privacy budget ε with respect to b^( h )_k is 

    ε( b_k^( h )) = ∑_i = 1^M_FΔ _i/√(∑_k = 1^M_P( w_ik b_k^( h ))^2[ ( 1 - p)^k - ( 1 - p)^M_P + 1])  ,

where  Δ_i is the sensitivity of the i-th element F̅_i^' of F̅', 1 ≤ i ≤M_F, w_ik is the weights between F̅_i^' and C̅_k^ * ^',  1 ≤ k ≤M_P, and p is the probability parameter greater than 0 and less than 1.

According to the chain rule, 

    ∂ε( b_k^( h ))/∂ b_k^( h ) = ∂ε( b_k^( h ))/∂ b_i^( F̅)( b_k^( h ))·∂ b_i^( F̅)( b_k^( h ))/∂ b_k^( h ) .

 Due to the independence of the elements of the feature vector,

    ∂ε( b_k^( h ))/∂ b_i^( F̅) =  - Δ _i/( b_i^( F̅))^2,

and

    ∂ b_i^( F̅)/∂ b_k^( h ) = b_k^( h )( w_ik)^2[ ( 1 - p)^k - ( 1 - p)^M_P + 1]/√(∑_k = 1^M_P( w_ik b_k^( h ))^2[ ( 1 - p)^k - ( 1 - p)^M_P + 1]).


Next, we substitute the above expression into the derivative ∂ε( b_k^( h ))/∂ b_k^( h ), and obtain it as follows

    ∂F_1( b_k^( h ))/∂ b_k^( h )
    
    =    | ∑_i = 1^M_FΔ _ib_k^( h )( w_ik)^2[ ( 1 - p)^k - ( 1 - p)^M_P + 1]/{∑_k = 1^M_P( w_ikb_k^( h ))^2[ ( 1 - p)^k - ( 1 - p)^M_P + 1]}^3/2|.
    
  






§ ANALYSIS OF PRIVACY AND UTILITY




Privacy. Now let's look at the privacy settings of the RDP for the feature vectors F̅' of the noisy face image according to Definition <ref>. In this paper, the elements of the feature vector are independent of each other with the given total privacy budget ε _0. 


RDP satisfies ε _0-differential privacy.






The face images A and B are a pair of neighboring face images. According to Definition <ref>, proving that RDP satisfies ε _0-differentially privacy is equivalent to prove 

    | [ F̅^( A )^']/[ F̅^( B )^']| ≤exp( ε _0),

where F̅^( A)^' and F̅^( B )^' is the noisy feature vectors of A and B. 




























Let F̅^( A̅) be the standard feature vector of a certain individual 𝔸's standard face A̅. Then, Δ _i is the sensitivity fot the i-th element of the noisy feature vector, and the Laplace scale parameter of that is b_i^( F̅). Define the privacy budget forthe i-th element of the noisy feature vector as ε _i, ε _i = Δ _i/b_i^( F̅), 1 ≤ i ≤M_F. Let ε _0 = ∑_i = 1^M_Fε_i, 

    | [ F̅^( A )^']/[ F̅^( B )^']| =    ∏_i = 1^M_F| [ F̅_i^( A )^']|/| [ F̅_i^( B )^']|
    
     =    ∏_i = 1^M_F| exp(  - | F̅_i^( A )^' - F̅_i^( A̅)|/b_i^( F̅))|/| exp(  - | F̅_i^( B )^' - F̅_i^( A̅)|/b_i^( F̅))|
    
     =    ∏_i = 1^M_F| exp( | F̅_i^( B )^' - F̅_i^( A̅)| - | F̅_i^( A )^' - F̅_i^( A̅)|/b_i^( F̅))|
    ≤   | ∏_i = 1^M_Fexp( | F̅_i^( B )^' - F̅_i^( A )^'|/b_i^( F̅))|
    ≤   | ∏_i = 1^M_Fexp( Δ _i/b_i^( F̅))|
    
     =    exp( ε _0)


Now we have

    | [ F̅^( A )^']/[ F̅^( B )^']| ≤exp( ε _0),

from which Theorem <ref> is proved.




Data utility. In this paper, we use the face image's PSNR (Peak Signal-to-Noise Ratio) value to represent the data utility ( visualization quality) of the face image. PSNR is a measure to evaluate image quality. The higher the value, the less the face image distortion. PSNR of the noisy face image is

    PSNR = 10 ·log _10( ( max{P̅'})^2/σ _r^2),

where P̅' is the one-dimensional pixel vector after the noisy pixel matrix P' flattened, max{P̅'} is the maximum element of P̅', and σ _r^2 is the real variance of P̅'. The higher PSNR is, the more image quality of the noisy face image retains, , the smaller the difference with the original face image. 






§ THE OPTIMIZED RDP ALGORITHM


Algorithm <ref> summarizes the optimized RDP: the inputs include the original face image pixel matrix P, the total privacy budget  ε _0, and the geometric superposition parameter p ∈ [0, 1]; and the outputs are the noise sanitized face image pixel matrix P' and the noisy feature vector F̅'.

First, we obtain the original sparse feature coefficient matrix C by multiscale feature dimensionality reduction of the original face image, which can be cast into the one-dimensional feature coefficient vector C̅. Then, the influence or weight w_ik the k-th element C̅_k^ * ^' of the sorted noisy wavelet coefficient vector C̅^ * ^', with respect to the i-th element F̅_i^' of the noisy feature vector F̅', is calculated, with 1 ≤ i ≤M_F, 1 ≤ k ≤M_P. After that, we obtain the optimized scale parameter b_k of the Laplacian noise for the k-th element C̅_k^ * ^' of the sorted noisy feature coefficient vector C̅^ * ^' according to the NA or the LMGD, with 1 ≤ k ≤M_P. Finally, the sorted noisy feature coefficient vector C̅^ * ^' is obtained through the Definition <ref>, from which we obtain the noisy feature coefficient matrix C' and the noisy feature vector F̅' of the noisy face image generated from C'. 




















Now let's look at the time complexity and storage complexity of Algorithm <ref>, which is mainly determined by two steps: 1) the step of calculating the weights between the noisy feature vector F̅' and the sorted noisy feature coefficient vector C̅^ * ^'; and 2) the step of obtaining the optimized Laplacian noise scale parameters of C̅^ * ^'.

Step 1) calculating the weights between the noisy feature vector F̅' and the sorted noisy feature coefficient vector C̅^ * ^' (lines 4-8): because F̅' has M_F element and C̅^ * ^' has M_P elements, the number of weights is M_F×M_P. Then the time complexity is 𝒪( M_FM_P) and the storage complexity of this stage is 𝒪( M_FM_P). 

Step 2) obtaining the optimized Laplacian noise scale parameters of C̅^ * ^' (lines 9-11): the number of elements in C̅^ * ^' is M_P. When the RDP is optimized by NA,  the temporal complexity is 𝒪( 1 ) and the spatial complexity is 𝒪( M_P). When the optimization is LMGD, the time complexity is 𝒪( M_P) and the spatial complexity is 𝒪( M_P). 
  
Since these two steps are sequentially composed, the time complexity is 𝒪( M_FM_P) and the spatial complexity is 𝒪( M_FM_P + M_P) if the scale parameters are optimized by NA; and the time complexity is 𝒪( M_FM_P + M_P) and the spatial complexity is 𝒪( M_FM_P + M_P) if the scale parameters are optimized by the LMGD. 


§ EXPERIMENTAL EVALUATIONS


We implement our simulations with Python 3.10 on a laptop with Intel Core i7-13700KF, 3.40GHz and Windows 10 system equipped with 64GB main memory.



 §.§ Experimental Setup




  §.§.§ Datasets


We use the following datasets to conduct experiments:
Labeled Faces in the Wild (LFW) <cit.> contains more than 13000 images of 5749 identities and provides a standard benchmark for face verification, composed of 6000 face pairs with 3000 matched and the other 3000 non-matched.
PubFig83 <cit.> contains 8300 cropped face images, made up of 100 images for each of 83 public figures.


  §.§.§ Evaluation Metrics


We use the following metrics in the experiment for performance comparisons: 

Variance. This metric measures the visualization quality of the noise sanitized face images. Because noise will have a negative impact on the quality of the face image, the smaller the variance of the face image's noise, the higher the retained quality of the face image. In this paper, we compare the theoretical variance and the true variance of the noisy image of five methods over the given range from ε_0 = 0.2 to ε_0 = 1.0 of the total privacy budget with probability parameter p=0.02. Since the noise follows the Laplace distribution, the theoretical variance σ _t^2= 2∑_k = 1^K ( b_k)^2, 




where b_k is the scale parameter of Laplacian noise of the k-th element C̅_k^ * ^' of the sorted feature coefficient vector C̅^ * ^' and K is the geometric number with mean 1/p.  

The real or actual variance during the numerical experiment, denoted as σ _r^2, is given by σ _r^2 = ∑_k = 1^M_P( P̅_k^'  - P̅_k)^2,



where P̅_k^' is the k-th element of the noisy one-dimensional pixel vector P̅' of the face image, and P̅_k is the k-th element of the original one-dimensional pixel vector P̅ of the face image.

Structural Similarity Index (SSIM) <cit.>. We conduct SSIM between the noise sanitized face images and the original face images. It imitates the human visual system and quantifies the attributes of images from brightness, contrast and structure. It uses the mean to estimate brightness, the variance to estimate contrast, and the covariance to estimate structural similarity.

Peak Signal-to-Noise Ratio (PSNR). This metric evaluates the visualization quality of the sanitized face images. It is calculated by Eq. (<ref>). The higher the PSNR value, the better the visualization quality of the sanitized face images or the less the face image distortion. 

False Negative Rate (FNR). This metric measures the effect of protection against the bypassing attack. It refers to the ratio of face images that should be identified as this individual but not. The larger the FNR is, the smaller the probability that the real facial features in this face image can be obtained, and the stronger protection for the method provides. 

In particular, for the accuracy of face recognition after adding noise, Ω ^( 𝔸) represents the face image dataset of the individual 𝔸, and F_𝔸 is a function that outputs the face recognition results, representing the feature vector of any face image A and the standard of 𝔸 whether the feature vector of the face satisfies Eq. (<ref>). If it does, the output is 1, , F_𝔸( A ) = 1, indicating that the result of face recognition is correct, otherwise the output is 0. f' represents the differential privacy mechanism with noise. Then, before adding noise, the accuracy is 1, that is [ . F_𝔸( A ) = 1|A ∈Ω ^( 𝔸)] = 1. Let [ . F_A( f'( A )) = 0|A ∈Ω ^( A )] = p̂, then 

    p̂ = ∏_i = 1^M_Fε _i/2Δ _i·exp(  - ∑_i = 1^M_Fε _i) =    ∏_i = 1^M_Fε _i/2Δ _i·exp(  - ε _0)
    ≤   ( ε _0/2)^M_Fexp(  - ε _0)/∏_i = 1^M_FΔ _i,

where ε _i^( F̅) is the privacy budget of the i-th element of the noisy feature vector. Thus, we can get 

    M_Fln( ε _0) - ε _0≥ln( p̂) + ∑_i = 1^M_Fln( Δ _i)  + M_Fln 2 .

From that, the total privacy budget ε _0 of the noisy feature vector has the ability to represent the accuracy of face recognition, , equal total privacy budget ε _0 can be considered as equal face recognition accuracy. Therefore, we will no longer separately show the comparison of the face recognition accuracy of the five methods on the two data in this paper. 



  §.§.§ Evaluated Methods

In experiments, we compare the proposed method with the existing advanced methods. And the methods are depicted as follows. 

DCT-DP. <cit.> proposed a privacy-preserving face recognition method based on frequency-domain Discrete Cosine Transform (DCT) differential privacy, , DCT-DP.

Pixel-DP. This method adds noises to all pixels of the face images. According to the geometric superposition, the Laplacian noises with the same scale parameter b̅_P are added to the selected geometric number of pixels of the face image, and the sum of the privacy budgets of the facial feature vector elements of the noisy image is the given total privacy budget.

RDP. This method is to add Laplacian noises with the same scale parameter b̅_W, as shown in Definition <ref>. 

RDP-NA. This method is the optimized RDP with NA proposed in Sec. <ref>). And its Laplacian noise with the optimized scale parameter b_k is calculated by Eq. (<ref>).

RDP-LMGD. The method is the optimized RDP through the LMGD proposed in Sec. <ref>). It calculates the optimized Laplacian noise scale parameter b_k by LMGD, with the step length or learning rate set to η∈ [0.01, 0.1].




 §.§ Evaluation on Privacy Budget and Utility



  §.§.§ Variance







	 












Now, let's look at the normalized difference or deviation between the real variance of numerical experiment σ_r^2 and the theoretical variance σ _t^2, which are shown in Figure <ref> and Figure <ref>. As the privacy budget ε_0 increased, for LFW data, the differences of the methods other than RDP (as shown in Definition <ref>) do not undergo drastic changes, but fluctuate within a narrow range; while the differences of the five methods consistently fluctuate within a small value range for PubFig83 data. Specifically, RDP-NA (RDP Optimized with Normalization Approximation, as seen in Sec.<ref>) and RDP-LMGD fluctuate slightly near the minimum value of 0 and do not exceed 0.05 for LFW data and PubFig83 data. This demonstrates the effectiveness of our defined cost function for the privacy budget allocation optimization problem. In addition, among the normalized differences between the other three methods, RDP fluctuates greatly, but is never higher than 0.15, while DCT-DP and Pixel-DP will never exceed ( 0.05, 0.15) range. Obviously, our experiment is reasonable and valid

























 






Finally, let's look at the real variance vs. the privacy budget ε_0. Figure <ref> and Figure <ref> show the fitted curves of the logarithm log(σ _r^2) of the real variance σ _r^2 of the five methods for the above two datasets under the given privacy budget. When the privacy budget ε_0=0.2,  the log(σ _r^2) of RDP-NA and RDP-LMGD on both data is less than 4.0. Meanwhile, log(σ _r^2)of DCT-DP is the smallest of the other three methods, around 4.75, yet it is significantly higher than that of RDP-NA and RDP-LMGD, which shows that the real variance σ _r^2 of RDP-NA and RDP-LMGD on the two data is significantly smaller than that of other methods when ε_0=0.2. Furthermore, as the privacy budget ε_0 increases, although the variance of all methods decreases significantly, no significant changes are found in this case, and log(σ _r^2)of RDP-LMGD is always slightly less than RDP-NA for LFW data, meaning that the real variance of RDP-LMGD sometimes is slightly less than RDP-NA. These findings suggest that RDP-NA and RDP-LMGD have less visual damage to the original face images than other methods.









  §.§.§ SSIM


In this paper, we conduct simulation experiments of SSIM on two real datasets for five methods within a given privacy budget range, as shown in Table <ref>. It can be seen that with the increase of privacy budget ε_0, RDP-NA and RDP-LMGD always have higher SSIM than other methods. This indicates that the scale parameters of Laplacian noise optimized by NA or LMGD can make the noisy image more similar to the original face image. In particular, the SSIM of RDP-LMGD is always slightly higher than that of RDP-NA in most cases, except that SSIM of RDP-NA in LFW data is slightly higher than that of RDP-LMGD when ε _0 = 0.2, and SSIM of RDP-NA and RDP-LMGD are equal when ε _0 = 1.0 on both data. 



  §.§.§ PSNR

Finally, let's look at the visualization quality of the sanitized face images after the sanitizing noise is added. 

In this paper, we compared the PSNR of the noise sanitized face images produced by the tested methods within a given range of total privacy budgets from ε_0 = 0.2 to ε_0 = 1.0 with the geometric superposition of parameter p=0.02. We obtained face images from two datasets within the given privacy budgets. The fitting curves of PSNR vs. face images are shown in Figure <ref>, from which we can observe that as the privacy budget increases, the PSNR becomes higher. As shown in Figure <ref>, the PSNR of RDP-NA and RDP-LMGD for LFW data higher than all compared methods. In particular, they are very close. Also, as the privacy budget increases, the advantages of RDP-NA and RDP-LMGD become more pronounced. Similarly, Figure <ref> shows the result For PubFig83 data, from which it is clear that the PSNRs of both RDP-NA and RDP-LMGD are much higher than those of compared  methods. For both data at high level of noise of ε_0 = 0.2, the PSNRs of both RDP-NA and RDP-LMGD significantly exceeded 50dB, while the best PSNR of the compared methods is only around 42dB, which is ∼10dB less than that of RDP-NA. In sum, both RDP-NA and RDP-LMGD achieve better visualization quality of the sanitized face images. 


 
Now, let's look at the visualization effect. As shown in Figure <ref> and Figure <ref>, we show the sanitized face images of five methods obtained  with a privacy budget ε_0 = 0.2 and the geometric superposition parameter of p=0.02 for LFW data and FubFig83 data. It can be shown that the Pixel-DP obtains more local mosaics on images than other wavelet transform based methods, which means that adding noise to selective wavelet coefficients of large significance on the privacy budget is better than adding noise directly to image pixels. Also, compared to the other four methods, the DCT-DP method contains deviation cross the whole face image, which is due to the global effect of the cosine basis used. On the contrary, no significant distortion nor visable mosaic effect can be seen in the sanitized face images of both RDP-NA and RDP-LMGD,  which means better data utility of visualization quality of the sanitized face images has been achieved.






 §.§ Evaluation of Bypass Attacks Against




We use the open-source facial recognition library face-recognition to calculate the Euclidean distance of the encoding vectors of the noisy facial image and the original face image. If the Euclidean distance is less than the set threshold 0.975, it is considered that the noisy facial image belongs to the certain individual to whom the original facial image corresponds. Otherwise, we consider it not the same face as the original facial image. 

Table <ref> shows the false negative rate of the face recognition corresponding to the five methods under the same utility settings on LFW data.





§ CONCLUSION

In this paper, a novel and efficient RDP is proposed to protect the facial features privacy for face image data, which can be formulated as the constraint optimization problem of maximizing the data utility or visualization quality of the noise sanitized face images, for a given privacy budget ε_0. First, the RDP calculates the influence or weights of the noisy feature coefficients in the multiscale transform subspaces with respect to the total privacy budget ε_0; then, the RDP adds Laplacian noises to the weight-ranked feature coefficients by geometric superposition, followed by the inverse transform to obtain the noise sanitized face image. It is rigorously proved that the noisy facial feature vector obtained with RDP satisfies ε_0-differential privacy. After that, the constraint optimization problem is formulated by the LM method by making the first-order Taylor expansion approximation on the influence or weight of the feature coefficients on the facial features. Furthermore, to solve the nonlinear LM, two methods are proposed, one for the real-time online applications and the other for the accurate offline applications: 1) the analytical NA method  for the real-time online applications: by assuming that all Laplacian noise scale parameters to be identical to their average value;  and 2) the LMGD method for the accurate offline applications: by computing the optimal scale parameters of Laplacian noise through iterative updating. Experiments on two real-world datasets are performed, which show that the RDP does achieve better data utility for a given facial feature privacy budget ε_0, compared to other state-of-the-art DP methods. 






IEEEtran


















 Paper Title: Vision-based Wearable Steering Assistance for People with Impaired Vision in Jogging.



Significant Contributions of the Higgs Mode and Self-Energy Corrections to Low-Frequency Complex Conductivity in DC-Biased Superconducting Devices
    Takayuki Kubo
    August 5, 2024
==================================================================================================================================================

empty
empty



§ ABSTRACT 
	
Outdoor sports pose a challenge for people with impaired vision. 
The demand for higher-speed mobility inspired us to develop a vision-based wearable steering assistance. 
To ensure broad applicability, we focused on a representative sports environment, the athletics track.
Our efforts centered on improving the speed and accuracy of perception, enhancing planning adaptability for the real world, and providing swift and safe assistance for people with impaired vision. In perception, we engineered a lightweight multitask network capable of simultaneously detecting track lines and obstacles. 

Additionally, due to the limitations of existing datasets for supporting multi-task detection in athletics tracks, we diligently collected and annotated a new dataset (MAT) containing 1000 images.
In planning, we integrated the methods of sampling and spline curves, addressing the planning challenges of curves. Meanwhile, we utilized the positions of the track lines and obstacles as constraints to guide people with impaired vision safely along the current track.
Our system is deployed on an embedded device, Jetson Orin NX. Through outdoor experiments, it demonstrated adaptability in different sports scenarios, assisting users in achieving free movement of 400-meter at an average speed of 1.34 m/s, meeting the level of normal people in jogging.
Our MAT dataset is publicly available from

<https://github.com/snoopy-l/MAT>








§ INTRODUCTION


According to statistics from the World Health Organization (WHO), nearly 253 million people worldwide are facing visual impairments <cit.>, which affects both their physical health and quality of life. 
Sports, like jogging, becomes particularly challenging for people with impaired vision, as they lack the visual cues essential for safe navigation. Even in Paralympic athletics, visually impaired athletes rely on guide runners, rendering independent running unattainable.

Advancements in artificial intelligence and wearable devices led to the development of assisting the blind and visually impaired (BVI) in overcoming common challenges in their daily lives<cit.>. For instance, <cit.> provides navigation speeds of 0.21 m/s, <cit.> facilitated walking speeds of 0.6 m/s, and <cit.> enables people with impaired vision to walk through hallways at speeds of 1 m/s.
These studies have contributed to enhancing the mobility of them. 
Nonetheless, there remains a dearth of exploration in sports and exercise, leaving the quest for faster independent mobility largely uncharted. Facilitating assistance for people with impaired vision in sports can also be beneficial for their physical health. Consequently, the design of wearable steering assistance tailored for people with impaired vision in sports holds crucial practical significance.

 

Based on visual sensors, we have designed a wearable steering assistance to guide people with impaired vision in independent movement within a representative sports scenario, the athletics track. The hardware of the assistance is illustrated in  Fig. <ref> and the overview is shown in Fig. <ref>. 
The perception and navigation capabilities developed within the athletics track hold the potential for broad applicability in sports, fitness, and competitions.
However, they also come with challenges.
Detecting the track lines presents difficulties because they are typically thin and can be easily obstructed by people ahead. Furthermore, there is an implicit correlation between the track lines and people obstacles. Additionally, the uniqueness of the scenario makes existing datasets cannot be directly used. The difference between curve and straight also poses a challenge for planning. 

To tackle these issues, we meticulously annotated an athletics track dataset for multitasks.
Meanwhile, we proposed a lightweight and efficient multitask network, enabling continuous track segmentation even in partial occlusion and the detection of people obstacles, bridging the gap between the two tasks. 
Moreover, we employ a combination of sampling and spline curves to effectively address the problem of curves. Our planner leverages the positions of track lines and obstacles as constraints to mitigate computational complexity. 
Even been deployed on embedded devices, our system operates efficiently, delivering swift and accurate directional instructions. Our main contributions are:


	
  * We designed a lightweight and efficient multitask network to achieve efficient and accurate obstacle detection and track line segmentation in embedded device. Additionally, we curated and annotated a dataset tailored for multitask in athletics track.
	
  * We developed a wearable steering assistance for people with impaired vision in jogging, enabling users achieving free movement of 400-meter at an average speed of 1.34 m/s, meeting the level of normal people.

 


§ RELATED WORKS

Based on visual sensors, many studies have developed devices suitable for people with impaired vision.

<cit.> uses visual and tactile feedback to assist the BVI in tasks of navigating mazes, locating chairs, and moving through crowds.
<cit.> use guidance robot to provide hospital navigation for people with impaired Vision.
<cit.> uses a visual positioning system to assist in indoor navigation for the BVI.
<cit.> employs an RGB-D camera to detect blind lanes and obstacles, using an improved artificial potential field method for navigation.
<cit.> can meet the needs of navigation, target localization, facial recognition, and text reading, attaining speeds of 0.21 m/s during navigation tasks.


<cit.> adds a camera to the white cane for object positioning, and combines multiple sensors to assist people with impaired vision in improving their walking speed of 0.6 m/s.
Relevant researches have greatly facilitated the lives of the BVI and improved their mobility.
But they cannot meet the demand for sports.
Assisting people with impaired vision in achieving higher-speed mobility within the athletics track needs to accomplish two essential tasks: environmental perception and path planning.

Detection and segmentation methods<cit.><cit.> can be employed for the perception of obstacles captured by RGB-D camera.
In terms of track line detection, we draw inspiration from lane detection in autonomous driving.


Deep learning-based lane detection methods include detection-based methods like LaneATT <cit.> and Ultra-Fast-Lane-Detection<cit.>. Segmentation-based methods include SCNN <cit.> and RESA<cit.>. While segmentation-based methods may struggle with occlusion issues, the detected lane lines are accurate for positioning. YOLOP<cit.> proposed using a single network to simultaneously perform tasks like traffic object detection, drivable area segmentation, and lane detection. YOLOPv2<cit.> further improves the performance and speed of each task. In terms of training data, there are many publicly available datasets in autonomous driving, such as BDD100K <cit.>, Cityscapes <cit.>. However, there is no dataset specifically designed for athletics track.

In terms of path planning, purely relying on a single method has its limitations, many research began to integrate different planning methods to achieve better results.
Baidu Apollo's EM planner <cit.> employs optimization function combined with polynomial curve for trajectory planning, utilizing dynamic programming and quadratic programming for path and speed planning respectively.<cit.> combines RRT and Dijkstra in path planning, where RRT is employed to reduce the planning area, while Dijkstra's optimizer is used to obtain the final output path. <cit.> combines graph-based search methods with sampling-based methods to quickly obtain the optimal solution.




§ METHODOLOGY





 §.§ System Overview
 
 
The overview of the assistance is shown in Fig. <ref>. 
The user is equipped with an RGB-D camera, which continuously captures real-time color and depth images. 
These inputs are processed within the perception module, which employs a multitask network with two decoder heads to perform track lines segmentation and obstacles detection, effectively establishing connections between the track and people.
In path planning module, we employ spline curves to facilitate the transformation between Frenet and Cartesian coordinates, addressing curve challenges.
Within Frenet coordinates, we use the track boundaries and obstacle positions as constraints for average sampling, reducing the sampling area and lowering the computational burden. We employ dynamic programming to calculate the minimum path cost between sampling points, yielding a locally optimal trajectory.
Finally, the path planning results are transformed into directional commands, guiding the user through voice feedback.



 §.§ Hardware Setup

 
As shown in Fig. <ref>, the wearable steering assistance comprises a pair of RGB-D glasses, an embedded device, and audio feedback. 
The wearable glasses are equipped with a RealSense D435i depth camera encased in a 3D-printed housing, enabling real-time capture of both RGB and depth images.
The embedded device is Jetson Orin NX, equipped with a processing capability of 100 TOPS.
This device is responsible for processing camera inputs and converting them into voice output.
The system is powered by a lithium battery and can operate continuously for 5 hours, meeting the endurance requirements for outdoor activities. 
The glasses themselves weigh 125g, while the embedded device weighs 274g and the overall weight is 1 kg, making it convenient to carry. 
The software is deployed on the Jetson Orin NX, eliminating the need for a remote server. The whole set of equipment fulfills the portability requirements.





 §.§ Dataset Construction




Given the uniqueness of the athletics track, we undertook the task of collecting, annotating, and curating an athletics track dataset, termed as Multi-data for Athletics Track (MAT), specifically tailored for multi-task detection. Our dataset comprises 1k track images, divided into training, evaluation, and test subsets at an 8:1:1 ratio. Each of these images was annotated with detection bounding boxes and track line masks. 
Some examples are shown in Fig. <ref>.
 

Our dataset features 2.47k instances of person. In terms of track line annotations, we have 376 images in straight and 624 images in curve. We performed continuous annotations on the track that were partial occlusion, resulting in 319 images with occlusions and 681 images without occlusion.











 §.§ Multi-task Network




We aim to achieve precise environment perception while operating on embedded devices. 
However, given the constraints of computational cost and memory, employing separate models for each task is impractical. 

To address this, we turn to multi-task networks, which leverage a shared backbone to extract valuable features and optimize performance across different tasks with separate decoders.
Therefore, we adopted an efficient and lightweight multi-task network capable of simultaneously handling track line segmentation and people detection tasks.

YOLOP <cit.> is capable of simultaneously performing tasks like traffic object detection, drivable area segmentation, and lane detection. However, it cannot be directly applied in the athletics track due to varying task requirements and speed constraints. To alleviate the challenge, we made a series of improvements to it.

Firstly, our tasks are different. We focus on people detection and track line segmentation for planning. Therefore, in addition to the data labeling efforts mentioned in section <ref>, we pruned down the detection head used for drivable area segmentation. 
The modified network now includes a shared encoder and two separate decoders for different tasks.
Secondly, we needed to meet the speed requirements for detection on embedded devices. Hence, we replaced the backbone with a lightweight architecture suitable for embedded devices, GhostNet <cit.>. Additionally, we replaced the spatial pyramid pooling module with spatial pyramid pooling-fast to reduce the computational load while maintaining performance. 
The above design may lead to a decrease in feature extraction capability. 
Consequently, we introduced the Coordinate Attention(CA) <cit.> at the end of the backbone to balance speed and accuracy without increasing computational demands.

In the detection head, we utilize FPN to propagate semantic features top-down and PAN for localization feature propagation bottom-up, achieving effective fusion.
In the track line segmentation head, we harnessed deep information from FPN. Through three upsampling processes, we acquire high-precision outputs, ensuring the continuity of track line predictions even in cases of partial occlusion.

we name our network RunnerNet, which substantially reduces the number of parameters and floating point operations compared to the original network. The network structure is shown in Fig. <ref>.



 §.§ Path Planning and Navigation







Given the track's unique characteristics, we employ an approach to combine averaged sampling with spline curves, which effectively leverages track information.


This strategy employs spline curves to convert track lines from Cartesian coordinates to Frenet coordinates, facilitating path planning within the Frenet coordinates. 

We confine the sampling region within the track lines, limiting the planning space and reducing computational complexity. 
Uniform sampling is then performed within this track region, and sample points are selected by loss function to yield locally optimal trajectory points.
Finally, the trajectory is transformed back to Cartesian coordinates. The yaw angle of the path is converted into voice output, providing guidance to users. 



  §.§.§ Utilizing cubic spline for the transformation between Frenet and Cartesian
 

Representing curves in Cartesian coordinates is challenging. By transforming the curved boundaries from the Cartesian coordinates to the Frenet coordinates, nonlinear constraints can be converted into linear constraints. However, directly utilizing formulas for conversion is complex.  
We employ cubic spline curves S_i(x) = a_i + b_ix + c_ix^2 + d_ix^3 for implicit solving. Unknown parameters a_i, b_i,  and d_i can be represented by c_i.

Performing cubic spline separately on the x and y coordinates, the curve can be uniquely represented as:

    S(t) = ( x(t),y(t)),

where S(t) represents the length of the curve in the Frenet coordinates, x(t) and y(t) is the cubic spline curves constructed under varying parameters. 
The arc length S and yaw angle ψ of the curve can be derived as:

    S = √(x'^2 + y'^2),


    ψ  = arctan (y'/x').

where x', y' is the derivative of x, y.
Given the length S_i, along with the segment interval i and interval parameters a_i, b_i, c_i, and d_i, it is possible to reverse the transformation from Frenet coordinates to Cartesian coordinates to obtain the  x and y coordinates. 



  §.§.§ Averaging sampling and cost function
 
EM planner<cit.> defines a reference line to connect global planning with local planning.
What sets us apart is that the detected track lines provide natural constraints for sampling. Utilizing spline curves, we represent the two boundaries of the track in the Frenet coordinates and incorporate the positions of obstacles. With knowledge of the track width and forward viewing distance, we uniformly sample n × m points both laterally and longitudinally.
While traditional average sampling requires large computation, the constraints of tracks significantly reduce the computational workload. 
Unlike road environments, average sampling and calculating cost function within the narrow space of the track can provide more comprehensive information.

After constructing the sampling space, each point could be evaluated by the summation of the cost function. The cost C_total between sample points can be expressed as:

    C_total = C_dis + C_obs,

where C_dis represents the distance cost between the current sampling point s_1 and the next sampling point s_2, and C_dis can be defined as:

    C_dis = . s_1 - s_2..

When there are obstacles, beyond the safe distance d_safe, the collision cost C_obs is inversely proportional to the collision distance d_col, the scale factor is k, C_obs can be defined as:

    C_obs = {[              0    no obstacle;        k/d_col d_col > d_safe;              ∞ d_col < d_safe ].


















We define the cost of the sampling points in the last row as the distance from the middle point.
Using the dynamic programming, we iterate the cost between sampling points from the last row forward. We set the initial point as the origin of the camera coordinates and iterate from front to back, selecting the sampling points with the minimum cost for each row as the trajectory points.
Finally, we use the cubic spline curve to fit the trajectory points to form a smooth path and obtain the yaw angle of the planned path by coordinate transformation.



  §.§.§ Audio Feedback
 
Effective feedback to the BVI is the key to achieving human-computer interaction and ensuring the availability of the system. We divided the visual field in front of the BVI into five sections corresponding to five direction instructions: forward, left forward, right forward, turn left, and turn right. The yaw angle of the planned output will fall into the corresponding interval, and the planning result will be converted into the direction instruction through voice feedback. In order to ensure real-time feedback, the trajectory planning is open-loop, and one direction instruction will be output for each frame.






































































§ EXPERIMENTS

In this section, we introduce the dataset setting, parameter setting, and experimental content. Experiments in section <ref> and section <ref> were carried out based on the configuration environment of NVIDIA GeForce RTX3060.
We ran all tests under the same experimental settings and evaluation metrics.
In section <ref>, we conducted outdoor experiments on the embedded device, Jetson Orin NX. 



 §.§ Settings

The BDD100K dataset <cit.> is used for multi-task detection in autonomous driving. Although the algorithms trained on the BDD100K dataset have strong transferability,
as shown in Fig. <ref>, due to the uniqueness of the athletics track, the model trained on BDD100K cannot be directly used for track.



We compared the detection performance of models trained on BDD100K, on our dataset MAT, and further fine-tuned on MAT after BDD100K's pre-traind. The results are presented in Table <ref>.
Utilizing BDD100K as a pre-trained model led to improvements in track line segmentation and yielded people detection results similar to models trained on our dataset. Consequently, we evaluate network performance by using models pre-trained on BDD100K and fine-tuned on MAT.

We resize images in BDD100K from 1280×720 to 640×384, and the images in MAT is 640×480.
In pre-training, we employ an Adam optimizer for 240 epochs with a 3-epoch warm-up, a batch size of 24, an initial learning rate of 0.001, and a weight decay of 0.0005. For fine-tuning, we train the models for an additional 240 epochs with a batch size of 24, and a learning rate of 0.001.








 §.§ Performance of Multitask Networks

The visualization of the network detection result is illustrated in  Fig. <ref>.
In section <ref>, we made improvements by pruning detection heads (-Head), changing the backbone (+GhostNet), and introducing attention mechanisms (RunnerNet). 
The impact of parameters and the performance of networks are shown in Table <ref>. 
We use Acc to evaluate the segmentation performance and use Recall and mAP_50 as the evaluation metric for detection accuracy.


Since the lack of prior works for multitask detection in the athletics track, we compared the performance of modified networks (-one head, +GhostNet, and RunnerNet).
The results indicate that our network reduced the FLOPs of the original network by 44% and improved the speed by 35%.
Although the modified model is smaller than the original network, it has achieved higher detection accuracy. 
RunnerNet achieved 92.6% accuracy in track line segmentation and achieved better detection performance compared to model (-Head). 
But its performance in detection is slightly lower than the model (+GhostNet), possibly due to its focus on the runway line.
To ensure the accuracy of track segmentation for the planning module, we choose RunnerNet, which performs best in track line segmentation, as the final model.

















































 §.§ Outdoor Experiments


To test the real-world usability of the system, three healthy subjects (User1, female, User2, male, and User3, male) were invited to wear the assistance for experiments, all participants wore blindfolds for the same level of vision.

In path planning, the sampling distance in the forward direction is 10 meters in Cartesian coordinates, with a sampling interval of 1 meter in Frenet coordinates and 5 sampling points in the horizontal direction. 

We categorized the sports scenarios into three situations:

	
  * Safe: no obstacle present, or obstacles have no impact.
	
  * Detour: detour around the obstacle and return.
	
  * Switch: change different track to bypass obstacles.


As shown in Fig. <ref>, we conducted tests to evaluate the users' performance in different sports scenarios.


Users were able to complete tasks of moving forward, detouring, and changing tracks with assistance.


In addition to testing their performance in various sports scenarios, subjects also wore the assistance for free movement experiments, including 100-meter straight, 100-meter curve, and  400-meter track. The number of experiments and average speeds are shown in Table <ref>.

The results indicate that, with the assistance of the device, users can achieve an average movement speed of 1.52 m/s in straight, 1.23 m/s in curve, and 1.34 m/s in track, reaching the level of normal people in jogging. Due to the processing speed limitations of embedded devices, we may not reach the speed of quick running, but it is sufficient to provide people with impaired vision with opportunities to participate in sports.




§ CONCLUSIONS


Based on visual sensors, we have developed a wearable steering assistance designed for people with impaired vision in jogging. 

We conducted outdoor experiments on the athletics track, and the results show that our wearable assistance enables the people with impaired vision to achieve an average speed of 1.34 m/s in track. The increase in speed is attributed to the accuracy of perception and the operational speed.
While our device provides faster mobility, it may not meet the speed requirements of quick running. In the future, we will continue to enhance the sports experience for people with impaired vision.

IEEEtran



-12cm   
                                  
                                  
                                  
                                  
                                  





















































 Paper Title: Add-SD: Rational Generation without Manual Reference.











1. Nanjing University of Science and Technology 

2. University of Adelaide 

3. VCIP, CS, Nankai University 

4. Baidu VIS 

† Equal contributions. ∗ Corresponding authors.






Add-SD: Rational Generation without Manual Reference
    
Lingfeng Yang^1† Xinyu Zhang^2,4† Xiang Li^3∗ Jinwen Chen^4 Kun Yao^4 Gang Zhang^4 Errui Ding^4 Lingqiao Liu^2 Jingdong Wang^4 Jian Yang^1∗

    August 5, 2024
=================================================================================================================================================



§ ABSTRACT 
Diffusion models have exhibited remarkable prowess in visual generalization. Building on this success, we introduce an instruction-based object addition pipeline, named Add-SD, which automatically inserts objects into realistic scenes with rational sizes and positions. Different from layout-conditioned methods, Add-SD is solely conditioned on simple text prompts rather than any other human-costly references like bounding boxes.
Our work contributes in three aspects: proposing a dataset containing numerous instructed image pairs; fine-tuning a diffusion model for rational generation; and generating synthetic data to boost downstream tasks. The first aspect involves creating a RemovalDataset consisting of original-edited image pairs with textual instructions, where an object has been removed from the original image while maintaining strong pixel consistency in the background. These data pairs are then used for fine-tuning the Stable Diffusion (SD) model. Subsequently, the pretrained Add-SD model allows for the insertion of expected objects into an image with good rationale. Additionally, we generate synthetic instances for downstream task datasets at scale, particularly for tail classes, to alleviate the long-tailed problem. Downstream tasks benefit from the enriched dataset with enhanced diversity and rationale.
Experiments on LVIS  demonstrate that Add-SD yields an improvement of 4.3 mAP on rare classes over the baseline. Code and models are available at <https://github.com/ylingfeng/Add-SD>.








§ INTRODUCTION

Generative models <cit.> have shown extraordinary development in vision generalization. Large pretrained diffusion models, , Stable Diffusion (SD) <cit.>, Imagen <cit.>, DALL-E <cit.> and SDXL <cit.>, provide powerful basis for various specific generation requirements, such as text-to-image editing <cit.>, image-to-image editing <cit.>, conditional generation <cit.>, and instruction tuning <cit.>, . Here, we classify the existing works into two categories, , algorithm-driven methods <cit.> and data-driven methods <cit.>.

Firstly, among algorithm-driven methods, Prompt-to-Prompt <cit.> and Plug-and-Play <cit.> utilize activation attention maps between text embeddings and noisy latent feature maps to guide content editing. GLIDE <cit.>, Blended Diffusion <cit.>, and DiffEdit <cit.> manipulate corresponding regions using either manually annotated semantic masks or predictive masks. These approaches excel at editing objects already present in the image. However, when tasked with adding unseen objects, they often struggle to generate desired objects convincingly.

Next, in terms of data-driven methods, Paint-by-Example <cit.>, GLIGEN <cit.>, AnyDoor <cit.>, X-Paste <cit.>, and Gen2Det <cit.> rely on predefined layouts with bounding boxes for image composition. This results in additional manual complexity and restricts layout diversity, thereby limiting generation variety and scalability. Other methods, like InstructPix2Pix <cit.> and MGIE <cit.>, use original-edited image pairs generated via Prompt-to-Prompt <cit.> to edit images with instructions. These approaches are more flexible, with instructions as text prompts being straightforward and avoiding costly layout design. However, it still faces difficulties ensuring consistency of the background content and the rationality of the newly added object.

Based on the literature, there is still no work that addresses the object addition task well, , inserting expected objects into realistic scenes with rational object sizes and positions without manual reference. To achieve this, we propose a simple yet effective pipeline named Add-SD. To ensure practicality and flexibility for human utilization, we follow InstructPix2Pix <cit.> by simply using instructions to add objects, which effectively reduces the human cost of layout design.

Add-SD consists of three essential stages to complete the object addition task: 1. creating image pairs by removing objects, 2. fine-tuning Add-SD, and 3. generating synthetic data for downstream tasks. The whole pipeline is depicted in Fig. <ref>.



The first stage aims to construct original-edited image pairs, which serve as the training data for the subsequent instruction-based diffusion model learning on rational object generation. Different previous methods which creates image pairs which are generated from paired-text, we apply reverse thinking by removing specified objects from an real image. This object removal operation ensures that the constructed image pairs have consistent backgrounds, and the removed objects provide the model with examples that satisfy visual plausibility and positional rationality (Fig. <ref>). Naturally, since removing and adding objects are inverse operations, the image after removing a specified object can be considered the “original” image, while the original image is used as the “edited” image in the object generation task. Our objective is to achieve consistent and rational image generations using instructions alone, eliminating the need for manual layout designs. To facilitate this, we develop various instruction templates, such as “Add a [obj].” or “Put the [obj] into the figure,” which are crafted manually or with support from ChatGPT <cit.>. The placeholder “[obj]” represents a detailed description of the expected object, encompassing its category, position, attributes, or relative relationships with other objects. This process results in the creation of RemovalDataset, which consists of original-edited image pairs accompanied by instructions.

The second stage involves training a diffusion model based on the created RemovalDataset, using the removal image and corresponding instruction as image and text conditions, respectively. The de-noising target is the original real image, , after adding the new objects. Using the above-produced original-edited image pairs, we follow InstructPix2Pix <cit.> to fine-tune a stable diffusion model, transforming it into an instruction-based addition stable diffusion model for the object addition task, denoted as the Add-SD model.
After the fine-tuning process, one can use our pretrained model to edit any image by adding objects. By prompting Add-SD with an original image and an instruction denoting the “[obj]” with descriptions of the objects expected to be inserted, the model performs the task. The visualization in Fig. <ref> demonstrates that our Add-SD can add the expected objects with rational positions and diverse variations.

In the last stage, we evaluate our Add-SD quantitatively on downstream tasks to show the effectiveness of generated images after adding new objects, as previous synthetic augmentation works <cit.> do. Initially, we design a super-label-based sampling strategy to decide which objects should be augmented. Then we employ the grounding detectors to localize the added objects in the generated images. This step also functions as a quality filtering process, wherein low-quality added objects with classification scores below a certain threshold are discarded. After that, we conduct object detection experiments using both real and generated images. Experiments with CenterNet2 <cit.> on COCO  and LVIS  show that Add-SD brings consistent improvements, especially 4.3 mAP increase on rare classes on the LVIS benchmark. Our Add-SD is a general and flexible method, in which each component can be replaced by state-of-the-art models.

In summary, our contributions are as follows:



  ∙ Introduce a flexible method Add-SD, an instruction-based addition pipeline designed to automatically insert objects into real images at rational positions. Add-SD simplifies the process by enabling the use of instructions for object addition without the need for costly human-layout design.


  ∙ Propose an instructed image pair creation strategy and form a RemovalDataset by removing objects from an image. This reverse thought process constructs original-edited image pairs, maintaining high visual plausibility and positional rationality.


  ∙ Improve the performance of object detection and instance segmentation with a generated synthetic dataset using our proposed Add-SD, where the added objects are localized and filtered by the existing grounding detector.




§ RELATED WORK




 §.§ Diffusion Models

Diffusion models for image generation have gained wide attention since the development of large pretrained models, including Stable Diffusion (SD) <cit.>, Imagen <cit.>, DALL-E <cit.>, and SDXL <cit.>. They are further integrated into specific applications such as personalized image generation <cit.> and local image editing <cit.>.

Personalized image generation methods perform poorly for creating new objects in images, as they generate diverse samples but lack consistency after editing. In local image editing, Prompt-to-prompt <cit.> uses guided attention between prompts and images for training-free editing, while Plug-and-Play <cit.> enhances attention by focusing on spatial features and self-affinities. However, these methods are limited by their attention guidance, focusing mainly on replacing instances rather than generating completely isolated targets.

Moreover, researchers have discovered that introducing prior masks instead of computed masks in Prompt-to-prompt shows results that better align with the expected generation positions. Blended Diffusion <cit.> employs a multi-step blending approach within the masked region to produce outputs with consistency. Inpaint-Anything <cit.> leverages SAM <cit.> to produce correlated masks. GLIGEN <cit.> handles grounding control by injecting grounding information into new trainable layers via a gated mechanism. 

Notably, in former works, the added objects are prompted with text instructions, while the image composition task designates the added target from given image conditions. Paint-by-Example <cit.> inpaints a semantically consistent object derived by CLIP <cit.> on the figure. 
TF-ICON <cit.> designs a training-free method to composite cross-domain images.
AnyDoor <cit.> proposes an ID extractor to enhance highly-faithful details. However, the mask-based methods are incompatible with tremendous generation subject to the limitation of requiring manual reference. In contrast, we aim to propose Rational Generation without Manual Reference.



 §.§ Instruction Tuning

This fine-tuning technique is harnessed to adapt pretrained models using human instructions, with the goal of enhancing usability. It finds extensive application in fine-tuning vision-language models <cit.>. Noteworthy examples include LLaVA <cit.>, which leverages visual instruction tuning through interleaved image-text dialogues with human instructions. Additionally, KOSMOS-2 <cit.> integrate bounding boxes into large multimodal models, enabling grounding capabilities.

In the realm of the vision generation <cit.>, InstructPix2Pix <cit.> introduces an image-to-image editing model, harnessing GPT-3 <cit.> and Prompt-to-Prompt <cit.> to construct a substantial synthetic dataset for instruction-based image editing. MagicBrush <cit.> manually constructs an annotated dataset for instruction-guided image editing. Emu Edit <cit.> employs multi-task training and proposes a matching architecture to address various generation tasks. Moreover, Emu2 <cit.> utilizes a large multimodal model to refine instruction prompting.

While instruction tuning heavily depends on training data pairs and prompted instructions, we highlight the constraints of the synthetic dataset constructed by Prompt-to-Prompt in InstructPix2Pix. Instead, we opt for an inverse operation of removing objects from the scene to generate superior data pairs with high consistency and rational distributions.



 §.§ Synthetic Data as Augmentation

Several studies have leveraged synthetic data as augmentation to enrich training samples and mitigate overfitting. In general classification tasks <cit.>, data generation prompts specific labels from the same domain as the evaluated benchmarks, such as ImageNet <cit.>, and long-tailed datasets like iNaturalist <cit.>. Generating images for classification is simpler than location-based tasks, primarily involving object-centric samples. However, for object detection or instance segmentation, the task complexity increases.

Cut-Paste-and-Learn <cit.> automatically cuts object instances and pastes them onto random backgrounds. Context-DA <cit.> improves pasting strategy by modeling surrounding visual context. Then, Copy-Paste <cit.> found that a simple strategy of randomly picking objects and pasting them at random locations on the target image could provide a significant boost, as long as specialized settings such as large-scale jittering are applied. While the former works use realistic instances cropped from natural images, X-Paste <cit.> inherits Copy-Paste's strategy but scales it up with more diverse samples generated by diffusion models.

However, these synthetic data struggle with whether to retain the pasted object. The heuristic pasting rule often ignores the rationality of added objects. There is a common understanding that placing training objects at unrealistic positions complicates implicitly modeling context, resulting in reduced detection accuracy <cit.>. Instead of complex designs, our Add-SD aims to learn rational addition solely from text instructions with a single diffusion model, offering efficiency and high variability.





§ ADD-SD


Our work focuses on the object addition task, , inserting isolated objects into real images in a rational manner. This includes believable object placement, natural object size, and seamless blending into the image background without altering the overall layout or obscuring existing foregrounds.

Existing image editing methods address the object addition task through structural designs like attention maps <cit.>, human reference layouts such as bounding boxes <cit.>, and instruction-based tuning with data pairs <cit.>. However, they encounter challenges in adding isolated objects, including the inability to independently add targets without reference objects, reliance on manually designed restrictions, and the lack of realistic image pairs that maintain positional and contextual consistency.

We thus propose Add-SD, an instruction-based pipeline specifically designed for the object addition task.
Add-SD can automatically add expected objects based solely on text prompts in a human instruction format, significantly reducing the need for reference layouts such as bounding boxes.
To display a rational generation process for objects' appearances, sizes, and positions, we perform Add-SD in three stages (Fig. <ref>): 1. creating image pairs by removing objects, 2. fine-tuning Add-SD, and 3. generating synthetic data for downstream tasks.



 §.§ Creating Image Pairs by Removing Objects

Image editing involves generating image pairs with consistent content but differing in local regions. Existing methods <cit.> first create text pairs with captions for before and after editing, then perform text-to-image generation to obtain synthetic image pairs. Although this approach is straightforward for creating instructed image pairs, ensuring consistency between the two images is challenging because both are generated by models.
To address this, we apply an inverse thinking process by removing existing objects from real images to create image pairs. In the object addition task, the images with removed objects are treated as the “original” images, while the real images are considered the “edited” images. This approach not only ensures the authenticity of the dataset, which aids in rational learning, but also provides fine-grained control over specifying the areas to be edited.

Here, we leverage LaMa <cit.>, an inpainting model, with parameters frozen for the object removal process.
Specifically, for each image I, we first obtain its instance mask M = { M_1, M_2, ..., M_k } from dataset, where k is the instance number.
If ground-truth masks are available, we use them directly; otherwise, we use the Segment Anything Model (SAM) <cit.> to extract masks from bounding boxes.
Then we use LaMa to remove each object, resulting in k images with removed objects, , R = LaMa(I, M), where R = { R_1, R_2, ..., R_k }.
Notably, one can remove multiple instances at a time to enhance the richness of the image pairs and facilitate the diffusion model's ability to generalize across multiple objects.

By performing the removal process, we construct an instructed image-pair dataset, termed RemovalDataset, with each R_i and I as original-edited image pairs and assign corresponding instructions P, such as “Add a [obj].” 
“[obj]” typically includes the instance category, as well as descriptions of position, attributes, or relative relationships with other objects.
To enhance user experience and usability, we extend the instruction format using ChatGPT <cit.> to generate various templates, such as “Put the [obj] into the figure.” This includes more complex sentences like “Integrate a [obj] into the scene for enhancement,” as well as interrogative forms like “Would you consider inserting a [obj] into the composition?” For scenarios involving the removal of multiple instances, corresponding instructions might be “Add two [obj].” or “Add [obj_1] and [obj_2].”

Now, we have constructed numerous original-edited image pairs (Fig. <ref>) using the object removal process, preparing them for the subsequent diffusion model fine-tuning. Notably, this image-pair creation pipeline theoretically allows for the generation of infinite samples without restrictions when incorporating existing tools. For instance, any region can be removed from the image, and detectors or large multimodal models can be used to extract object classes or scene descriptions as part of the instruction. For ease of quantitative assessment, we initially create samples using a detection dataset with instances annotated with ground truth descriptions and masks as a pioneering experiment.





 §.§ Fine-tuning Add-SD

In this stage, we fine-tune a diffusion model to learn rational object addition from the above-constructed RemovalDataset. We choose to use the recently popular stable diffusion model <cit.> as the generative model.
Our model is built upon InstructPix2Pix <cit.>, , fine-tuning a pretrained SD model conditioned on given image and text instructions.
For an input image I, we denote its latent embedding from the VAE encoder ℰ as z=ℰ(I).
The image condition is the latent embedding of the removal image R after inputting into ℰ, , c_I=ℰ(R).
The text condition is derived by embedding the instructions: c_T=𝐓𝐞𝐱𝐭𝐄𝐧𝐜(P).
Then noise ϵ sampled from 𝒩(0, 1) is added to z for t times to obtain the noise latent z_t:

    x_t = √(α̅_t) x + √(1 - α̅_t)ϵ,

where α̅_t is calculated from the accumulated variance schedule and t is sampled from the overall time-steps T. Next, a U-Net <cit.> with parameters ϵ_θ is utilized to predict the added noise.
ϵ_θ is updated via the following optimization objective:

    ℒ = ϵ - ϵ_θ(z_t, t, c_I, c_T)^2.

Following former practice <cit.>, we initialize the weights of our model with a pretrained Stable Diffusion checkpoint to maintain its vast text-to-image generation ability.
We do not start our training from InstructPix2Pix pretrained checkpoints, since it is trained on limited synthetic data, which may not align well with images in real scenes and may be insufficient for the object addition task.

After training, the updated model can generate completely new objects in a real image given an adding instruction. For example, if one wants to add a dog to the image, just input “Add a dog in the image.” This process is very simple and easy to use. 
The RemovalDataset creation stage and the fine-tuning stage complete the main model of visual generation, , Add-SD. In the next stage, we will use Add-SD to generate a new dataset for downstream tasks to test how these generated images influence performance as a quantitative evaluation.



 §.§ Generating Synthetic Data for Downstream Tasks

Synthetic data augmentation has been verified as a valuable technique to improve performance in various vision tasks. Since our Add-SD is able to add expected objects, we evaluate its capability on object detection and instance segmentation downstream tasks, especially addressing the long-tail problem. 
Following existing works <cit.>, we generate synthetic data based on COCO <cit.> and LVIS <cit.>. Importantly, we consider the following three aspects to generate our new synthetic data and combine it with vanilla data for joint training on detection (Fig. <ref>).
Finally, we provide a discussion on the structural comparison between Add-SD and existing works of synthetic data augmentation (Fig. <ref>).

Which objects should be added?
Our Add-SD can flexibly insert specified objects using only text instructions. Therefore, we focus on adding more objects that appear less frequently in the dataset to mitigate the negative effects of the long-tail problem. However, certain categories may be inappropriate for a given image. For instance, it would be implausible to add a dog to an image of a plane in the sky or to include a snowboard in a scene with a traffic light. To address this issue, we propose a method of collecting the super-categories of existing instances within an image, as depicted in Fig. <ref> (b). We then sample labels that fall under the same super-category. This approach ensures that the selected category is generated in a rational manner, in principle.
We then sample from the filtered classes, as shown in Fig. <ref> (c). Specifically, for COCO, we randomly sample from all classes, with each weight being the reciprocal of the number of label instances. For LVIS, we uniformly sample from the categories officially defined as rare classes. For each image, we add one or multiple rare instances to expand the training sample set.





What types of synthetic images should be used?
After inserting the expected objects into real images, we obtain various synthetic images. These synthetic images cannot be used for training detectors directly, as we lack annotations for the newly added objects. To address this, we use a grounding method <cit.> to locate the added instances, using their labels as queries (see Fig. <ref> (e)). 
Notably, the queries include only the class names of the added objects, not all instances in the image. For the original instances, which are expected to be preserved in the synthetic images, we directly copy all ground-truth boxes from the real images into the generated ones.
Additionally, the detection score from grounding results could serves as a data filter, discarding added objects with scores below a specified threshold. We also tested several data filtering strategies from previous works, such as using aesthetic scores or CLIP scores to filter out poor-quality images <cit.>, using classification scores to remove poor-quality instances originally present in the real images <cit.>, and deleting objects that are either too small or too large <cit.>. However, these methods did not yield further improvements. Therefore, we have decided to not use any additional filtering strategies for simplicity.
Examples of the generated dataset are depicted in Fig. <ref>.

How to train the generated data?
Directly mixing the generated data with the original data using a weighted sampler, , mixing generated images and original images with a specific ratio in a batch, brings limited enhancement. This is because we do not predict masks for the synthetic data; the mask loss is only computed on the real data. This situation causes instability in the gradient updating of mask losses, as it requires multiplying a binary mask to disable calculations on generated samples without a ground truth mask. As a result, we follow Gen2Det <cit.> and use a batch sampling strategy instead, , sampling the entire batch with either original data or synthetic data with a pre-defined threshold. This approach avoids additional processing on the mask loss when the current batch is sampled from synthetic data.

Structural comparison with existing works.
The synthetic data pipeline of ours and other methods is depicted in Fig. <ref>. Copy-Paste <cit.> and X-Paste <cit.> both require a segmentor to filter out masks of the target to be pasted. Their difference lies only in whether the target is cropped from a real image or generated by Stable Diffusion <cit.>. Then, they are pasted onto another real image with strategies such as random pasting or choosing the content with more empty space. Their pipeline is complex and thus time-consuming during the training of detectors. Also, as depicted in Fig. <ref> (a), the bicycle may be placed over the lake or in the sky. On the contrary, our method, once trained on an Add-SD with RemovalDataset, requires only one step to generate instances onto it with randomly sampled labels as instructions. Therefore, in Fig. <ref> (b), the bicycle is generated on the pier near the boat with a reasonable size and orientation.





§ EXPERIMENTS

In this section, we elaborate on the evaluation of Add-SD in terms of image-editing metrics and downstream tasks. First, we summarize the datasets and implementation details of the employed architecture. Then, we evaluate image editing performance through a user study and several generation metrics, comparing it to other commonly used image editing methods. Next, we compare the downstream performance of models trained with our generated data against other synthetic augmentation methods. Additionally, we conduct ablation studies on the components of Add-SD. Finally, we provide qualitative analysis through visualization comparisons.



 §.§ Datasets

To develop a diffusion model with strong generalization performance, we collect datasets from various fields, including Detection, Visual Question Answering, and Referring Expression Comprehension. The datasets we use are COCO <cit.>, LVIS <cit.>, Visual Genome <cit.>, RefCOCO <cit.>, RefCOCO+ <cit.>, and RefCOCOg <cit.>.

COCO is a widely-used dataset for object detection and segmentation tasks, with over 200,000 images across 80 categories. LVIS expands on COCO with 1,000+ categories and intentionally imbalanced annotations, offering a more challenging benchmark. Visual Genome provides more detailed annotations, including object relationships and attributes, across 100,000+ images. RefCOCO is specialized for referring expression comprehension, offering images with textual descriptions.

In the context of downstream tasks, to ensure fairness, we exclusively utilize COCO as the data source for creating RemovalDataset and generating based on COCO, and similarly for LVIS. Conversely, we train the ultimate Add-SD Model by leveraging all datasets mentioned above, as Visual Genome and RefCOCO offer richer instructions in addition.





 §.§ Implementation Details

For training Add-SD, we follow InstructPix2Pix <cit.> to fine-tune on RemovalDataset for maximum 50,000 iterations. During inference, we set the diffusion steps N to 100 by default. For downstream tasks, we use the settings of X-Paste <cit.>, running 4× scripts with a batch size of 8 per GPU and a total of 90,000 iterations by default. We utilize CenterNet2 <cit.>, Mask R-CNN <cit.>, and Faster R-CNN <cit.> with ResNet-50 <cit.>, pretrained on the ImageNet-22K, as the detectors. 
For the ablation study, we use CenterNet2 on COCO as the default setting.
The codebase is built upon Detectron2 <cit.> and X-Paste <cit.>. All experiments are conducted on 8× Tesla V100 GPUs.







 §.§ Image Editing Evaluation


User Study.
We generate 100 samples each for InstructPix2Pix <cit.>, MagicBrush <cit.>, and our Add-SD under the same instructions and turn to multiple users (∼10) for blind testing, gathering ratings in terms of visual quality, rationality, and consistency. The quality metric emphasizes the overall visual appeal and fidelity of the generated images. Rationality assesses whether the generated objects make sense or are rational in the visual context, including object size, position, orientation, . Finally, consistency measures how well the background content and overall context are preserved before and after editing. We report the user preference ratio (%). The results in Table <ref> show that our method obtains a major preference in human evaluation.

Quantitative metrics for the image editing.
We employ editing metrics of L1, L2, CLIP-I, CLIP-T, and DINO. L1 and L2 measure pixel-wise distance between generated and ground-truth images from the COCO dataset. CLIP-I and DINO gauge similarity between image features extracted by CLIP and DINO, respectively. CLIP-T evaluates feature similarity between image and text using text input “Add a [obj]”. Our method outperforms others across most metrics (Table <ref>). Notably, CLIP-T metric has a bias toward only specifying the added object in text, rather than the entire content of the generated image.





 §.§ Comparisons on Downstream Tasks


Different synthetic data augmentation methods.
We compare the detection and segmentation AP scores between detectors trained with our synthetic data and others using the CenterNet2 architecture for object detection (Box AP) and instance segmentation (Mask AP). We first conduct experiments on the LVIS <cit.> dataset. Note that the official results reported in X-Paste are irreproducible, as stated in the open issue [<https://github.com/yoctta/XPaste/issues/2>] and the appendix of Gen2Det.
We re-run the LVIS experiments following the exact configuration file of the X-Paste codebase, which aligns with the reproduction results within Gen2Det.
From Table <ref>, we observe that, specifically for the rare classes, our method surpasses the baseline by 4 box AP and X-Paste by 3.8 box AP. Overall, our method demonstrates improvements in box AP/mask AP by +1.0/+0.8 compared to the baseline.
Similarly, Table <ref> shows the comparison of various methods on COCO dataset.
Among the methods listed, Add-SD stands out as the most effective, achieving the highest scores for both Box AP and Mask AP. This suggests that the Add-SD method offers significant improvements over existing techniques in terms of accuracy for both object detection and instance segmentation tasks, benefiting from the augmented synthetic datasets. 
Additionally, as shown in Fig. <ref>, the rare classes generated through Add-SD help alleviate the long-tail problem, as the increased number of rare classes leads to a higher growth rate.
Consequently, Add-SD presents a promising approach for enhancing performance in these domains when utilizing the CenterNet2 architecture on the LVIS and COCO datasets.










Different detectors.
To verify the transferability of generated datasets, we additionally compare our methods against the baseline of Faster R-CNN <cit.>, Mask R-CNN <cit.>, and CenterNet2 <cit.> benchmarked with COCO. We use the codebase and configs in Detectron2 <cit.> with a 3× schedule using ResNet-50 and FPN settings. From Table <ref>, we observe a consistent improvement of Add-SD over the baseline, indicating the robustness of our generalized data. Notably, on CenterNet2, it achieves the largest performance boost of 0.85 points.



 §.§ Ablation Study

Synthetic data sampling ratio.
Using complete generation data for training may cause severe performance drop due to a wide distribution change from the real images. A common solution is to mix the synthetic data and real data for joint training. Therefore, we specifically explore the mixture ratio between the two datasets. The ratio represents the proportion of generated images to all training samples, where a ratio of 0 equals the baseline and a ratio of 1 means using synthetic data only. 
Results in Table <ref> indicate that a sampling ratio of 20% yields the best performance for mixed learning on COCO. To maintain consistency, the same ratio is applied to the LVIS dataset.
Notably, a mixture ratio of 50% still improves the baseline by 0.5 box AP, demonstrating the effectiveness of the generated training data.

Batch sampling.
In this section, we ablate the sampling strategy. Table <ref> compares the performance of two sampling strategies: “Normal Mixture” and “Batch Sampling.” The “Normal Mixture” strategy samples both real and synthetic images within a single batch, while the “Batch Sampling” strategy samples the two types of images in separate batches.
Across the evaluation on COCO using CenterNet2, “Batch Sampling” demonstrates better results, suggesting that it is a viable enhancement strategy. This approach potentially offers benefits in terms of computational efficiency or ease of optimization.

Training schedule.
We explore the influence of the training schedule on performance by extending the training time from 4× to 8× scripts. Table <ref> shows that our method achieves continuous improvement with the extended training time. Benefiting from the synthetic data, the detector can prevent overfitting and boost performance in scaled-up settings. Notably, Gen2Det <cit.> generates 5× more instances than ours. However, by training for 2× longer, we can achieve better performance than Gen2Det (47.22 47.18 box AP).









Super-label selection strategy.
In order to generate samples for rare classes to alleviate the long-tailed problem, we prompt Add-SD with labels under label-wise frequency calculated from COCO. Moreover, we observe that certain labels cannot be generated for specific figures. For example, it is difficult to generate a dog in an image capturing a plane in the sky. Therefore, we propose a label selection strategy by selecting labels in the range of the super-labels of the original instances in the images. In this case, the generated instances and the original instances in the image exhibit relatively similar characteristics, reducing the occurrence of unreasonably generated instances. Table <ref> shows that using super-labels demonstrates an effective enhancement in detection performance by generating more reasonable data samples.





Scale up the synthetic data.
To indicate the generative diversity of Add-SD, we scale the total number of generated samples from 100K to 200K images. Notably, our total number of iterations remains unchanged (90K) to ensure a fair comparison.
Table <ref> illustrates the scalability of synthetic data augmentation with samples generated by Add-SD in downstream tasks, highlighting the benefits derived from diverse training samples.



 §.§ Qualitative Results

Compare to different image editing methods.
In this section, we compare the visualization of the editing results between our Add-SD and other image editing methods including InstructPix2Pix <cit.>, and MagicBrush <cit.>.  
From Fig. <ref>, we observe that our method maintains background consistency while providing diverse generalization. For example, in the first image, we generate dogs with different poses and create fire hydrants with various locations and colors, whereas other methods produce similar samples. Additionally, MagicBrush tends to replace existing objects with generated ones, resulting in the spoon covering the original fork. Furthermore, existing methods fail to ensure rationality in the position of the bicycle or the sizes of the dog.

Learn from instruction.
Since we use datasets like RefCOCO and Visual Genome, which include descriptive statements as instructions, the diffusion model can learn about object properties, orientations, and their relative relationships with surrounding objects. This enables more controlled and rational generation. By managing the instruction inputs, we can produce more diverse images with reasonable placement of new objects.
For each image in Fig. <ref>, we design different instructions to show the generation diversity of Add-SD. The first line shows the relation awareness of Add-SD, which can distinguish the surrounding objects in complex environments such as the living room. The second sample shows Add-SD can handle attributes as inputs for generating more detailed instances. The example shown in the third line demonstrates a perspective relationship, with near objects appearing larger and distant objects appearing smaller. It also illustrates the ability to discern positional relationships between objects. Finally, we show that Add-SD can generate different numbers of targets by prompting with direct numbers or quantity descriptors. It is notable that when we prompt with “Add 2 giraffes”, roughly 3/4 of the images (with a total generation of 50 trials) successfully generated two objects. However, in other cases, it's still possible to generate one or more instances, indicating that instructions regarding quantity remain a challenging issue to address.


Limitations.
There are some limitations to Add-SD. Since the training datasets are generated using an inpainting model, they can be affected by cases where the object is not removed naturally, such as residual blur or shadows (see Fig. <ref>). Additionally, processing large targets is more challenging, as removing a large instance can result in unsatisfactory inpainting, which reduces dataset quality.



§ CONCLUSION

We propose Add-SD, a novel visual generation method for instruction-based object addition, demonstrating significant advancements in seamlessly integrating objects into realistic scenes using only textual instructions. By employing object removal, we create instructed image pairs and construct a dataset called RemovalDataset. We then train an instruction-based stable diffusion (SD) model on this dataset to learn diverse and rational object generation without relying on costly human annotations such as bounding boxes.
The fine-tuned Add-SD enhances image generation quality, rationality, and background consistency, as verified through user studies and various editing metrics. Additionally, we design a super-label sampling strategy to generate synthetic samples to boost downstream tasks. Evaluation on the LVIS  datasets reveals a notable improvement of 4.3/1.0 AP points on rare classes/overall compared to the baseline, highlighting the effectiveness of Add-SD. This approach holds promise for applications in various domains requiring automatic scene augmentation.



§ ACKNOWLEDGMENT

The authors would like to thank the editor and the anonymous reviewers for their critical and constructive comments and suggestions. 
This work is supported by
the National Science Fund of China under Grant No. 62361166670,
the Young Scientists Fund of the National Natural Science Foundation of China (Grant No. 62206134), 
the Fundamental Research Funds for the Central Universities (070-63233084), 
and the Tianjin Key Laboratory of Visual Computing and Intelligent Perception (VCIP). 
Note that Lingfeng Yang and Jian Yang are with 
PCA Lab, Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, and Jiangsu Key Lab of Image and Video Understanding for Social Security, School of Computer Science and Engineering, Nanjing University of Science and Technology.



§ BROADER IMPACTS

Further research and careful consideration are necessary when utilizing this technology, as the presented proposed method relies on statistics derived from training datasets that may possess biases and could potentially result in negative societal impacts.



§ DATA AVAILABILITY STATEMENT

We confirm that the data supporting the findings of this study are available within published articles of COCO <cit.>, LVIS <cit.>, Visual Genome <cit.>, RefCOCO <cit.>, RefCOCO+ <cit.>, and RefCOCOg <cit.>.


unsrt




 Paper Title: SmileyNet -- Towards the Prediction of the Lottery by Reading Tea Leaves with AI.




.jpg,.pdf,.mps,.png

The Bulk Densities of Small Solar System Bodies as a Probe of Planetesimal Formation
    [
    August 5, 2024
====================================================================================



§ ABSTRACT 
We introduce SmileyNet, a novel neural network with psychic abilities. It is inspired by the fact that a positive mood can lead to improved cognitive capabilities including classification tasks. The network is hence presented in a first phase with smileys and an encouraging loss function is defined to bias it into a good mood. SmileyNet is then used to forecast the flipping of a coin based on an established method of Tasseology, namely by reading tea leaves. Training and testing in this second phase are done with a high-fidelity simulation based on real-world pixels sampled from a professional tea-reading cup. SmileyNet has an amazing accuracy of 72% to correctly predict the flip of a coin. Resnet-34, respectively YOLOv5 achieve only 49%, respectively 53%. It is then shown how multiple SmileyNets can be combined to win the lottery.     






§ INTRODUCTION

Modern AI is one of the greatest if not the greatest scientific breakthrough of all times. The publication of the structure of DNA by Watson and Crick in Nature <cit.> has only achieved slightly more than 19,000 citations by summer 2024 according to GoogleScholar. And it has taken 71 years. AI <cit.> has led to many important papers, which received many more citations within a much shorter time-frame; see, e.g., Tab. <ref> for a few examples based on citations according to GoogleScholar.



Given the relevance of AI for our modern world, it is surprising that there are areas of high economic relevance where it has not been applied, yet. We explore here - up to our knowledge for the very first time - the potential of AI for the psychic services industry, which is a multi-billion dollar business. This industry is estimated to have reached a revenue of well above $2 billion in the USA in 2023. And it experiences a steady further growth. This holds especially for psychic online services that managed an exponential growth in recent years according to market-reports.       

We suspect that AI may already be used in some real-world psychic services, especially in the online branches. But the methods and the results are not academically published up to our knowledge - likely due to the large economic potential they possess. While this is somewhat understandable, we believe in Open Science <cit.> and we share our insights due to the large benefits Psychic AI can provide to humanity.

A first medium-term goal is to win the lottery to further fund research in the novel field of Psychic AI. A well-established method of Tasseology (also known as Tasseomancy or Tasseography) <cit.> is used to this end, namely reading tea leaves <cit.>. In the spirit of divide and conquer, the simpler problem of predicting the outcome of flipping a coin is tackled first. We then show how this can be used to predict the numbers for winning the lottery jackpot. 

As an important basic research result, it is proven in this paper, up to our knowledge for the very first time, that a neural network can possess psychic powers. Concretely, it is shown that the novel SmileyNet can predict the flipping of a coin with a precision well above chance. 




     


§ SMILEYNET


SmileyNet is a novel bio-inspired network. It is motivated by the fact that cognitive processes can be positively influenced by a positive mood 
<cit.>.
This includes especially also classification tasks <cit.>. 



Furthermore, it is well-known that visual stimuli are well suited to elicit emotions <cit.>. This includes in particular a smiling face <cit.>. We hence first present the network with smileys as input to bias it into a good mood. More precisely, the Unicode emoticon "U+1F642" for the classical smiley, officially known as "Slightly Smiling Face" is used to this end (Fig.<ref>).

But stress is a negative factor for cognition <cit.>. We do not want to loose the positive effects of the good mood by putting the network under stress in the smiley training phase. Hence, it gets positive feedback <cit.> that it is doing fine, i.e., that it is doing nothing wrong.

More precisely, the training in the smileyfication phase is done as follows. A quadratic loss function L_smile() is used. To avoid putting the network under stress and to give a further positive endorsement, the expected output ŷ is set to be the actual output y whenever a smiley is presented to the net, i.e., ŷ = f_I(y) where f_I() is the identity function. So,


    L_smile()= 1/2 (y - ŷ)^2  with ŷ = f_I(y)


This method of smileyfication can be applied to any network N_old to turn it into a novel one N_new, which is expected to perform better than the previous N_old. 

Often, there may only be small improvements and it can happen that multiple trials are needed until some improvement is seen for the first time, which should then be immediately documented. Also, it can be useful to take additional factors into account. For example, the effects of smileyfication may only be seen on a sunny day as, e.g., the weather has an influence on the mood <cit.>.  



§ RELATED WORK


YOLOv5 <cit.> and Resnet34 <cit.> are well-known networks for image classification. They are used for comparison to SmileyNet in Sec. <ref>. There are also many other networks. But as YOLOv5 and Resnet32 are very popular, we concentrate here on those two. 





YOLOv5 is the fifth iteration of the revolutionary "You Only Look Once" object detection model. It is designed to deliver high-speed, high-accuracy results in real-time. Built on PyTorch, this powerful deep learning framework has garnered immense popularity for its versatility, ease of use, and high performance. Fig. <ref> shows how it works.


ResNet-34 is a convolutional neural network in PyTorch. It is has 34 layers. It addresses the problem of vanishing gradient. This is done with the identity shortcut connection that skips one or more layers. 
Fig. <ref> shows how it works.




§ HIGH-FIDELITY SIMULATION


The constellation of tea leaves and the predicted event are linked by cosmic energy. We use this fact to generate high-fidelity training data. This is not easy in the context of psychic powers and it may be one reason why AI has not been used - up to our knowledge - in this economically highly relevant business up to now.

The core idea of our approach for training data is to use the reverse direction of the link. The coin is flipped in a deterministic way, i.e., it is so to say turned by hand in rounds r from one side to the other. The related constellation of tea-leaves, i.e., the image I_r is then generated afterwards. 

In the software implementation, "heads" is presented with a "1" and "tails" with a "0" for the sake of convenience and wlog. Also wlog, we start with "tails" facing up as initialization of the flipping process. In the very first round r=1, the coin is turned and it hence shows "heads". The coin is then turned again and it now shows "tails" in round r=2, and so on. More formally, it holds for the outcome o_flip(r) in round r that


    o_flip(r) = 
    heads    if  (r+1)  mod  2 = 0
    tails    otherwise



The images I_r in our high-fidelity simulation are based on a real-world image (Fig. <ref>)[wikimedia.org; curid=17659832; user: MochaSwirl; Public Domain] of tea-leaves in a cup used by a fortune-telling professional. From this professional image, real-world pixels p_tea belonging to tea and real-world pixels p_cup belonging to the cup are extracted. This forms the base image I_0 with 100 × 100 pixels (Fig. <ref>). For the sake of simplicity, p_tea-pixels are shown in black and white is used for p_cup-pixels . 




The first image I_1 is then generated right after the coin was flipped for the first time. So, I_1 is linked to producing "head" according to the laws of Tasseology. In general, each image I_r is generated from I_r-1 by randomly selecting k_change coordinates [x,y]_j : x,y ∈{0,...,99} that are altered as follows 


    I_r[x][y] = 
    
        p_tea   if  I_r-1[x][y] = p_cup
    
        p_cup   if  I_r-1[x][y] = p_tea











Each I_r is generated right after doing the flip of the coin in round r, i.e., it is linked to the outcome o_flip(r) of flipping the coin in that round. According to our experience, small to medium values for k_change work best. The process is illustrated in Fig. <ref>. As mentioned, the p_cup, respectively p_tea pixel are shown as white, respectively black pixel in Fig. <ref> for the sake of simplicity. 



§ EXPERIMENTS AND RESULTS

  
The high-fidelity simulation presented in Sec. <ref> is used to generate 500 images. When it is needed by the network, the images are scaled to fit the requirements of PyTorch. The set is split into the first 300 images that are used for training and the following 200 images are used for validation. 


 


Tab. <ref> shows the results for YOLOv5 and Resnet-34. Resnet-34 has only 98 out of 200 cases correctly predicted. YOLOv5 does a bit better and it manages 106 correct predictions on the testing set.

Due to the better performance of YOLOv5 compared to Resnet-34, it is used for a smileyfication to generate SmileyNet version 1. As SmileyNet is expected to have an improved performance, it is trained with less data. Only every fifth image from the data-set is used. Note that this is still an alternating sequence of "heads" and "tails", i.e., it is exactly the same principle as before, but just 100 training images. The remaining 400 images are used for testing.

SmileyNet manages an amazing accuracy of 71.75%, i.e., 287 out of 400 cases are correctly predicted. This is well above the performance of YOLOv5 and Resnet-34 (Tab. <ref>). Most importantly, it is well above chance and we therefore prove - up to our knowledge - for the very first time that AI can possess psychic powers. Note that it can be shown that the improvement is statistically significant, i.e., it is a thorough and sound result.       

Tab. <ref> summarizes the performance gains compared to YOLOv5 and Resnet-34. SmileyNet has an accuracy of 71.75%, i.e., there is an increase by 35.38% from the 53% of YOLOv5. The performance gain is in the comparison with Resnet-34 with 46.43% even higher. SmileyNet requires only 100 training images while YOLOv5 and Resnet-34 had 300 images. So, there is a reduction by 200 images or 66% in comparison to the other two nets. 



§ SMILEYNET FOR LOTTERY FORECASTS



The forecasting of throwing a coin can already be a useful psychic power. But the forecast of lotteries has a very straightforward economical potential. Lotteries exist in most countries around the globe with high jackpots for the correctly predicted numbers. It is shown in the following with the example of the German lottery how SmileyNet can be used to this end.

In the German lottery, six numbers win_1,...,win_6 are drawn out of a pool with the numbers 1,...,49. The psychic power of SmileyNet can forecast the flip of a coin, i.e., 1 bit. We therefore simply combine multiple SmileyNets to encode the numbers 1,...,49. Then, one after the other win_i is predicted. 

More precisely, let b_j denote the forecasting output of SmileyNet N_j and ⟨ b_1,...,b_n ⟩_2 denotes the number x encoded by the bits b_1,...,b_n, i.e.,

    x = ⟨ b_1,...,b_n ⟩_2 = ∑_i=1^nb_i · 2^n-i


The i-th number win_i of a lottery draw is then predicted with ten SmileyNets N_j with outputs b_j as follows 

    win_i = ⟨ b_1,...,b_5 ⟩_2 + ⟨ b_6,...,b_9 ⟩_2 + ⟨ b_10⟩_2


As mentioned, the winning numbers win_1,...,win_6 are simply generated one after the other. It must of course be ensured that fresh images of tea-leaves from a just emptied cup are provided in every round. Nonetheless, SmileyNet does not have perfect psychic power. It may hence happen in rare occasions that a number win_j+k has been predicted before as win_j. But this is not a problem. Then, the current win_j+k is simply discarded and an other round is added to determine a new win_j+k.

Given the predictive power of SmileyNet to correctly forecast a bit with p_ N=0.7175, the exact likelihood p_win of forecasting the German lottery with the above method can be computed as

    p_win   = p_ N^5 ·p_ N^4 · p_ N
       = 0.03615922 
       = 3.615922 · 10^-2
        
This likelihood may look small at first glance, but it is by six orders of magnitude better than just chance with

    p_chance   = 49  6^-1
       = (49!/6!(49-6)!)^-1
       = 1/13,983,816
       = 7.15112 · 10^-8
       ≪ p_win
        

This is an important result that demonstrates the huge potential of the novel field of Psychic AI. Note that it is a thorough result as it can be shown that it is statistically significant.

But before we apply these results in practice, we still want to improve the performance of our approach. This holds first and foremost for the likelihood p_win of predicting the correct numbers for winning the jackpot:

	
  * One option is simply more training data and more computation power for a larger network to turn SmileyNet into a foundation model for psychic AI. 

  * Furthermore, we expect a transformer to be more accurate given that the drawing of lottery numbers is a weekly event in Germany. This is left for future work.

Also, a reduction of the required number of SmileyNets would be nice before applying it in practice as it is a bit inconvenient to drink so much tea. 



§ CONCLUSIONS


SmileyNet version 1 was introduced. It is based on smileyfication, i.e., the fact that a positive mood can have a positive influence on cognitive functions including classification. It was shown how SmileyNet can be used to predict the lottery by reading tea-leaves. First, the flipping of a coin was predicted with a very good accuracy, namely 72%. State of the art networks like YOLOv5 and Resnet-34 perform much worse. Second, it was shown how multiple SmileyNets can be combined for lottery forecasts. In addition to the potential practical implications, it is a major breakthrough to show that AI can have psychic powers.  



§ ACKNOWLEDGMENTS

We thank our anonymous reviewers for their feedback. Special thanks go to reviewer 2, who pointed out that the drawing of lottery numbers tends to be a weekly or at least regular event, and that the use of a transformer should be beneficial. 




§ EPILOG

This is a satirical accumulation of misconceptions, mistakes, and flawed reasoning I have encountered in recent times as a reviewer and sometimes even as a reader of published papers. I hope it is entertaining and useful in the context of the education of BSc, MSc, and PhD students in Machine Learning, Artificial Intelligence, and Cognitive Science. 

For an easy start, here a few questions to reflect upon. First and foremost, there is of course the following puzzle: 

	
  * Where does the "magic" hide that is presented here? How can SmileyNet "predict" the flip of a coin better than chance? 

As a small hint: there is a single conceptual mistake that explains it. 

There are also many other bugs, respectively misconceptions and flawed reasoning in this paper. Here some further food for thought, to hopefully teach a good lesson by setting a bad example:   

	
  * Can AI beat natural laws? What are the pros and cons of a detailed pre-analysis of the possible information content of the data?
	
  * How much academic novelty is in SmileyNet? How much does it differ from the net it is build upon? 
	
  * How do we prevent smileyfication in our own work? How can we properly show that an "extension" or "modification" of an existing approach leads to a substantial improvement and study the related trade-offs?  
	
  * What is a proper assessment of the state of the art? How do we find, understand, and discuss related work?
	
  * What is a proper statistical performance analysis? What is statistical significance and how should it be interpreted?
	
  * When and how do I expect my own work to be of relevance? Why do I publish it? 
	
  * What are Artificial Intelligence and Machine Learning? Which sub-areas within these disciplines exist? What are commonly used methodologies within the sub-areas? 
		
  * What is the role of training data in supervised machine learning? What are the pros and cons of simulation?
	
  * What is Open Science? What are its principles? What does it imply for the quality of the underlying work? What are the pros and cons if others, respectively oneself adheres to it? 
	
  * What is a bio-inspired design? How can insights from Cognitive Science be used and which pitfalls exist?
	
  * What are the pros and cons of bibliometrics? How can they help us to identify relevant work? And where are their constraints and limitations?

This list is a bit concentrated on the "big questions". It is of course also possible to just concentrate on the (many) methodological mistakes in the SmileyNet paper. So, the above list is neither meant to be exhaustive nor to be without dispensable aspects depending on the context. It is just a possible starting point for using the SmileyNet in the class-room, or to just meditate about it on your own, or... 

Enjoy and have fun...

10

url@samestyle








WatsonCrick-DNA-Nature1953
J. D. Watson and F. H. C. Crick, “Molecular structure of nucleic acids: A
  structure for deoxyribose nucleic acid,” Nature, vol. 171, no. 4356,
  pp. 737–738, 1953. [Online]. Available:
  <https://doi.org/10.1038/171737a0>


DeepLearning-Hinton-Nature15
Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521,
  no. 7553, pp. 436–444, 2015. [Online]. Available:
  <https://doi.org/10.1038/nature14539>


DeepLearning-Book-MITpress16
I. Goodfellow, Y. Bengio, and A. Courville, Deep learning.1em
  plus 0.5em minus 0.4emMIT press, 2016.

DeepLearning-ResNet-CVPR16
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image
  recognition,” in IEEE Conference on Computer Vision and Pattern
  Recognition (CVPR), 2016, Conference Proceedings, pp. 770–778.

Adam-StochasticOptimization-arXiv2014
J. B. Diederik P Kingma, “Adam: A method for stochastic optimization,”
  arXiv preprint, vol. arXiv:1412.6980, 2014.

DeepLearning-AlexNet-NIPS12
A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with
  deep convolutional neural networks,” pp. 1106–1114, 2012. [Online].
  Available:
  <http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks>


Breimann-RandomForests-ML01
L. Breiman, “Random forests,” Machine Learning, vol. 45, no. 1, pp.
  5–32, 2001. [Online]. Available:
  <https://doi.org/10.1023/A:1010933404324>


DeepCNN-LargeScale-ImageRecognition-2014
K. Simonyan and A. Zisserman, “Very deep convolutional networks for
  large-scale image recognition,” arXiv preprint, vol. arXiv:1409.1556,
  2014.

GAN-NIPS2014
I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair,
  A. Courville, and Y. Bengio, “Generative adversarial nets,” 2014. [Online].
  Available:
  <https://proceedings.neurips.cc/paper_files/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf>


FasterR-CNN-NIPS2015
S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time
  object detection with region proposal networks,” 2015. [Online]. Available:
  <https://proceedings.neurips.cc/paper_files/paper/2015/file/14bfa6bb14875e45bba028a21ed38046-Paper.pdf>


ImageNet-CVPR09
J. Deng, W. Dong, R. Socher, L. J. Li, L. Kai, and F.-F. Li, “Imagenet: A
  large-scale hierarchical image database,” in IEEE Conference on
  Computer Vision and Pattern Recognition (CVPR), 2009, Conference
  Proceedings, pp. 248–255.

DeepLearning-GoogLeNet-CVPR15
C. Szegedy, L. Wei, J. Yangqing, P. Sermanet, S. Reed, D. Anguelov, D. Erhan,
  V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in
  IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
  2015, Conference Proceedings, pp. 1–9.

UNESCO-RecommendationOpenScience2021
UNESCO, “Unesco recommendation on open science,” 2021. [Online]. Available:
  <https://doi.org/10.54677/MNMH8546>


Tasseology-2008
R. E. Guiley, “Tasseomancy,” in The Encyclopedia of Witches, Witchcraft
  and Wicca.1em plus 0.5em minus 0.4emCheckmark Books, 2008.

Tasseology-TeaLeaves-2016
K. Henriott-Jauw, Tea and Tasseomancy: A Guide to Reading Tea
  Leaves.1em plus 0.5em minus 0.4emCreateSpace Independent
  Publishing Platform, 2016.

Tasseology-TeaLeaves-2014
A. O'Reilly, Tea with Annie, a divinitive guide to Tasseomancy and its
  artistry.1em plus 0.5em minus 0.4emMelbourne, Australia:
  Whiteslaw Press, 2014.

PositiveMood-CreativeProblemSolving-Sage2015
J. Politis and J. C. Houtz, “Effects of positive mood on generative and
  evaluative thinking in creative problem solving,” Sage Open, vol. 5,
  no. 2, p. 2158244015592679, 2015. [Online]. Available:
  <https://journals.sagepub.com/doi/abs/10.1177/2158244015592679>


PositiveMood-CognitiveControl-2011
E. A. Martin and J. G. Kerns, “The influence of positive mood on different
  aspects of cognitive control,” Cognition and Emotion, vol. 25, no. 2,
  pp. 265–279, 2011. [Online]. Available:
  <https://doi.org/10.1080/02699931.2010.491652>


PositiveMood-Cognition-Classification-2010
R. T. Nadler, R. Rabi, and J. P. Minda, “Better mood and better
  performance:learning rule-described categories is enhanced by positive
  mood,” Psychological Science, vol. 21, no. 12, pp. 1770–1776, 2010.
  [Online]. Available:
  <https://journals.sagepub.com/doi/abs/10.1177/0956797610387441>


ImagesMoodSurvey2016
M. K. Uhrig, N. Trautmann, U. Baumgärtner, R.-D. Treede, F. Henrich,
  W. Hiller, and S. Marschall, “Emotion elicitation: A comparison of pictures
  and films,” Frontiers in Psychology, vol. 7, 2016. [Online].
  Available:
  <https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2016.00180>


ImagesMoodSurvey2008
J. K. Olofsson, S. Nordin, H. Sequeira, and J. Polich, “Affective picture
  processing: An integrative review of erp findings,” Biological
  Psychology, vol. 77, no. 3, pp. 247–265, 2008. [Online]. Available:
  <https://www.sciencedirect.com/science/article/pii/S0301051107001913>


MoodSmilingFace2019
J. Nikitin and A. M. Freund, “The motivational power of the happy face,”
  Brain Sciences, vol. 9, no. 1, p. 6, 2019. [Online]. Available:
  <https://www.mdpi.com/2076-3425/9/1/6>


CognitionStressNegative2024
G. Picciotto and R. A. Fabio, “Does stress induction affect cognitive
  performance or avoidance of cognitive effort?” Stress and Health,
  vol. 40, no. 1, p. e3280, 2024. [Online]. Available:
  <https://onlinelibrary.wiley.com/doi/abs/10.1002/smi.3280>


CognitionStressNegative2023
E. J. Kim and J. J. Kim, “Neurocognitive effects of stress: a metaparadigm
  perspective,” Molecular Psychiatry, vol. 28, no. 7, pp. 2750–2763,
  2023. [Online]. Available: <https://doi.org/10.1038/s41380-023-01986-4>


StressCognitionFeedback2020
M. Paul, C. Bellebaum, M. Ghio, B. Suchan, and O. T. Wolf, “Stress effects on
  learning and feedback-related neural activity depend on feedback delay,”
  Psychophysiology, vol. 57, no. 2, p. e13471, 2020. [Online].
  Available: <https://onlinelibrary.wiley.com/doi/abs/10.1111/psyp.13471>


MoodWeather2008
J. J. A. Denissen, L. Butalid, L. Penke, and M. A. G. van Aken, “The effects
  of weather on daily mood: A multilevel approach,” Emotion, vol. 8,
  no. 5, pp. 662–667, 2008.

MoodWeather1984
E. Howarth and M. S. Hoffman, “A multidimensional approach to the relationship
  between mood and weather,” British Journal of Psychology, vol. 75,
  no. 1, pp. 15–23, 1984. [Online]. Available:
  <https://bpspsychub.onlinelibrary.wiley.com/doi/abs/10.1111/j.2044-8295.1984.tb02785.x>


YOLOv5-2021
Ultralytics, “YOLOv5: A state-of-the-art real-time object detection
  system,” 2021. [Online]. Available: <https://docs.ultralytics.com>


YOLOv5-architecture-2021
——, “Ultralytics YOLOv5 architecture,” 2021. [Online]. Available:
  <https://docs.ultralytics.com/yolov5/tutorials/architecture_description/>






 Paper Title: Identity-Consistent Diffusion Network for Grading Knee Osteoarthritis Progression in Radiographic Imaging.





 



IC-RDN for Grading Knee Osteoarthritis Progression






W. Wu et al.




The University of Sydney, Darlington NSW 2008, Australia 

{wenhua.wu, kun.hu, wenxi.yue, weiwilson.li, milena.simic, zhiyong.wang}@sydney.edu.au Sydney Polytechnic Institute Pty Ltd, Sydney, NSW, Australia

chris@ruddergroup.com.au School of Computing, Engineering & Mathematical Sciences, La Trobe University

w.xiang@latrobe.edu.au
Identity-Consistent Diffusion Network for Grading Knee Osteoarthritis Progression in Radiographic Imaging
    Wenhua Wu10000-0001-7488-5487 Kun Hu1,Corresponding author. 0000-0002-6891-8059 
Wenxi Yue 10000-0002-8270-417X Wei Li 10000-0003-4731-3226 Milena Simic 1 Changyang Li 20000-0001-9400-3998 Wei Xiang 30000-0001-9400-3998 Zhiyong Wang 10000-0002-8043-0312 

    August 5, 2024
===================================================================================================================================================================================================================================================================



§ ABSTRACT 
Knee osteoarthritis (KOA), a common form of arthritis that causes physical disability, has become increasingly prevalent in society. Employing computer-aided techniques to automatically assess the severity and progression of KOA can greatly benefit KOA treatment and disease management. Particularly, the advancement of X-ray technology in KOA demonstrates its potential for this purpose. Yet, existing X-ray prognosis research generally yields a singular progression severity grade, overlooking the potential visual changes for understanding and explaining the progression outcome. 
Therefore, in this study, a novel generative model is proposed, namely Identity-Consistent Radiographic Diffusion Network (IC-RDN), for multifaceted KOA prognosis encompassing a predicted future knee X-ray scan conditioned on the baseline scan. 
Specifically, an identity prior module for the diffusion and a downstream generation-guided progression prediction module are introduced. 
Compared to conventional image-to-image generative models, identity priors regularize and guide the diffusion to focus more on the clinical nuances of the prognosis based on a contrastive learning strategy.  
The progression prediction module utilizes both forecasted and baseline knee scans, and a more comprehensive formulation of KOA severity progression grading is expected.  
Extensive experiments on a widely used public dataset, OAI, demonstrate the effectiveness of the proposed method. 

  




§ INTRODUCTION





Knee osteoarthritis (KOA) is a common form of arthritis, mainly affecting knee joints with the symptoms of knee joint swelling, tenderness, and bone spurs. As the major factor of physical disability, KOA has recently shown an increasing prevalence in society, especially among the elders <cit.>. Currently, there are rare treatments for curing KOA without physical surgery (total knee replacement) <cit.>. Therefore, to relieve the pain and control the progression of KOA, the diagnosis and prognosis of the disease are significant, using different clinical assessment methods, such as medical imaging scans and patients' clinic questionnaires. In specific, the imaging scans are utilized to detect the structural changes within the knee joint areas, which indicates the characteristics of KOA. Typically, the severity of KOA is measured by the Kellgren and Lawrence (KL) grading scale system, which encompasses five different levels, from grade 0 (healthy cases) to grade 4 (highly severe cases) <cit.>. 

Compared to the extensive time and human resources required for specialists to detect changes, employing computer-aided techniques to automatically assess the severity and progression of KOA can significantly reduce time costs. This aids in the prompt diagnosis and prognosis of KOA at early stages <cit.>. 
Methods for grading the severity and progression of KOA have evolved from traditional machine learning approaches <cit.>  to advanced deep learning techniques <cit.>.  These modern approaches utilize various forms of input, including single images, multiple images, and images combined with demographic information (such as age, gender, BMI, and the WOMAC questionnaire) <cit.>.
Particularly, the advancement of X-ray technology and its application in KOA significantly improves the capability to detect and monitor the progression  <cit.>. Meanwhile, utilising the generative AI methods in the KOA analysis field focuses on augmenting the knee radiological dataset to improve the grading process <cit.>. 
Yet, existing X-ray prognosis research generally yields a singular progression severity grade using the baseline X-ray scan <cit.>, overlooking the potential visual changes for understanding and explaining the progression outcome. However, the visually predicted outcome is also crucial in clinical practice to explain the disease trajectory. 

In addition to enhancing these methods, we hope the longitude visual content will also aid clinicians in their decision-making by providing further explainability and as domain knowledge to guide the algorithms. For the methodology part, previous findings revealed that GANs yield poor bone structures and inconsistency in patients' identity information, which motivated us to devise a diffusion-based method with identity information. Identity patterns are crucial with a principle widely utilized in the forensic field. We anticipate that incorporating such information improves the consistency between baselines and synthesized longitude images.

Therefore, in this study, a novel deep generative model, namely Identity-Consistent Radiographic Diffusion Network (IC-RDN), is proposed for multifaceted KOA prognosis. IC-RDN creates a predicted future knee X-ray scan conditioned on the baseline scan and a future KOA severity grade as illustrated in Fig. <ref>. 
Specifically, an identity prior module for guiding the diffusion and a downstream generation-guided progression prediction module are introduced. 
Compared to a conventional image-to-image generative model,  identity priors regularize and guide the diffusion to focus more on the clinical nuances related to the prognosis based on a contrastive learning strategy.  
Using both forecasted and baseline knee scans, we anticipate a more comprehensive and enhanced formulation of KOA severity progression measured in grades with the progression prediction module.  
Extensive experiments on a widely used public dataset, OAI <cit.>, demonstrate the effectiveness of the proposed method. 




The key contributions of this study can be summarised in three-fold:


 
  * A novel deep generative model - IC-RDN is proposed for multifaceted KOA prognosis prediction for the first time, encompassing radiographic visual predictions and grading forecasting. 
 
  * An identity prior module for persisting patient's identity information on bone X-ray images is proposed to guide the diffusion process focusing on clinical nuances. 
 
  * Comprehensive experiments with the public OAI dataset demonstrate the effectiveness of the proposed method. 




§ RELATED WORKS




 §.§ Computer-Aid KOA Severity Grading



Existing studies in grading the severity of KOA can be categorised into KOA detection and KOA multi-class grading <cit.>. The KOA detection focuses on categorising several or combined severity grades from the whole grading system to achieve a coarse-grained classification. 

For instance, a graph neural network has been designed to utilize the shape boundary information of knee bones for KOA detection <cit.>. 
In addition to shape, texture features have been independently utilized for KOA detection to examine the internal pixel relationship <cit.>. 
To leverage both textural and shape information, the features extracted from the shallow and deeper layers of a DenseNet model are combined, where the shallow layers contain more shape patterns and the deeper layers focus on textural information <cit.>.

Multiclass grading, on the other hand, concentrates more on exploring each grading severity level in the system. 
As the KL grades can be considered a continuous measurement, with convolution neural networks, both regression and classification strategies are studied <cit.>. 
Likewise, an ordinal loss function is proposed to formulate the continuous nature of KL grades <cit.>. 
In parallel, the attention mechanisms identify the significant areas related to KOA multiclass grading <cit.>. 
In <cit.>, attention mechanisms are studied to combine and fuse multi-scale features from a network. 
Furthermore, to take advantage of the symmetrical structure of knee joints, a Siamese Network is employed to investigate the relationship between both knee joints. This approach facilitates the analysis of intricate KOA patterns in both fully supervised and semi-supervised manners, addressing the challenge posed by the scarcity of annotated datasets <cit.>. Likewise, a self-supervised method has been introduced. This method leverages both X-ray and MRI images to expand the pool of positive pairs in contrastive learning settings, offering a robust solution to dataset limitations <cit.>. 



 §.§ Computer-Aid KOA Longitude Progression Grading
 


Apart from KOA severity grading, predicting the progression of the disease is also significant for medical treatment. Existing works mainly focus on different aspects in predicting the progression of KOA: 1) Knee joint pain status; 2) KOA risk factor prediction; 3) Joint space narrowing (JSN) and cartilage structural change prediction; 4) KL grade prediction; and 5) total knee replacement surgery (TKR) prediction. 

Early studies on grading the progression of KOA mainly rely on conventional machine learning algorithms. In specific, the feature selection is leveraged in filtering the significant features contributing to the progression grades <cit.>. Parallelly, the comparison among different machine learning classifiers is conducted to select the algorithm with better progression grading performance <cit.>. <cit.> utilises the XGBoost model to predict the knee joint width as a KOA progression factor. To take advantage of various classification algorithms, the ensemble strategy is deployed among RandomForest, logistic regression, and support vector machine <cit.>. 
These machine learning-based methods show some encouraging results in feature engineering, where the extracted features are combined and utilized with methods such as multilayer perceptrons (MLPs) to predict the cartilage volume from MRI scans <cit.>. 

With the powered deep learning methods explored in both general and medical fields, end-to-end approaches have shown promising results in KOA progression grading. An MLP is deployed as a classifier to predict progression grades <cit.>. Neural Networks, otherwise, are utilised to predict pain status via extracting information from radiographics and demographics <cit.>. Likewise, CNN-based methods are also leveraged in analysing various KOA risk factors <cit.> and predicting the JSN progression <cit.>. To explore the KOA evolutionary trajectory, an Adversarial Evolving Neural Network is proposed to involve the estimated KOA patterns at different grades as auxiliary information <cit.>. 
Although predicting the direct KOA symptoms shows a clearer progression measurement, grading the progression KL grades and predicting TKR are also able to illustrate the progression in the clinical assessment domain. A binary classification model is designed based on whether the patient needs to take TKR as a future event <cit.>. 

Despite using medical imaging only, the information from multiple modalities, including imaging scans, clinical questionnaires, and demographics, are leveraged in a multimodal method to analyse the KOA progression comprehensively <cit.>. However, these studies only produce a direct outcome for progression grading ignoring the visual information from the given medical imaging techniques. Therefore, including visual information as an additional modality for analysing comprehensive multifaceted progression patterns is attractive to be explored. 






§ METHODOLOGY
 

As shown in Fig. <ref>, the proposed IC-RDN consists of three modules: 1) an identity prior module extracts patient's identity representations; 2) a diffusion-based generative module produces the future X-ray images based on the current-stage X-ray scans and the associated identity representations; and 3) a down-stream generation-guided progression prediction module that predicts the progression severity using both current and generated future X-ray images to provide comprehensive multifaceted progression patterns. 



 §.§ KOA Progression Radiographic Generation


This study incorporates X-ray medical imaging scans to predict the progression of KOA. The objective is to forecast the severity of KOA progression from two perspectives, encompassing severity grading score and visual information (i.e., a generated knee X-ray scan in the future - 12-month in this study).  
Initially, radiographic images for joints are captured during the baseline (first) X-ray scan, referred to as the baseline visit or 0-month visit. Patients are then asked to undergo a follow-up X-ray scan after 12 months for longitudinal tracking.  
Specifically, the i^th longitude X-ray joint pairs from the dataset is denoted as 𝐈_visit^i ∈{𝐈_baseline^i, 𝐈_12-month^i }, where 𝐈_visit^i ∈ℝ^C× W× H and C, W and H are the number of channels, the width and the height of the X-ray joint images. For notation simplicity,  𝐈_Baseline^i  and 𝐈_12 months^i can be shortened as 𝐈_v00m^i  and 𝐈_v12m^i. And the generated 12-month X-ray image for the i^th joint pair is denoted as 𝐈̂_12 months^i or 𝐈̂_v12m^i. The corresponding severity grade for the 12-month visit is indicated by y_KL^i and the grades produced by the prediction model are shown by ŷ_KL^i. 





 §.§ Diffusion-based Visual Prognosis


The diffusion-based generative module in the IC-RDN is designed to produce 12-month X-ray images from baseline visit images for understanding and explaining the potential visual changes in terms of KOA patterns. 
This incorporates a T-step Denoising Diffusion Probabilistic Model (DDPM) <cit.> under an image-to-image transition manner. 
The DDPM consists of two primary processes: the forward (or diffusion) process and the reverse process. 
Typically, in the forward process, the original data x_0 ∼ q_data(x_0) is transformed towards an isotropic Gaussian distribution. 
However, within our image-to-image diffusion model, the aim is to map the data x_0 ∼ q_data(x_0) to x_T, where, in our context, x_0 represents a 12-month visit X-ray image 𝐈_v12m^i, and  x_T corresponds to the baseline visit X-ray images 𝐈_v00m^i. 
This transformation is modelled as a fixed Markov Chain of length T. 
Subsequently, the reverse process mirrors the forward process, aiming to predict  
x_0 from x_T ∼ q_data(x_T). 

However, executing both processes directly in pixel space is highly resource-intensive, necessitating substantial memory and computational power. To mitigate this, a VQ-GAN-based encoder (see Fig. <ref> (b.1.)) encodes x_0 into a latent space z_0, wherein both the forward and reverse processes (see Fig. <ref> (b.3.)) are performed to generate z_T. A VQ-GAN-based decoder (depicted in Fig. <ref> (b.2.)) then decodes  z_T back from the latent space to the pixel space (x_T). The objective of this stable diffusion-based approach <cit.> is formulated as follows:

    ℒ_LDM := 𝔼_ε(x), ε∼𝒩(0,1), t[ ε - ε_θ(z_t, t) ^2_2 ],

where ε_θ(z_t, t) denotes a denoising autoencoders (time-conditional UNet) and θ contains the corresponding weights, where t = 1...T. 



 §.§ Knee Joint Identity Prior and Injection


To persist a subject's identity information in generating the corresponding 12-month X-ray scan, an identity prior module is devised to extract the identity-sensitive patterns. 
Specifically, we denote the identity representation for the i^th subject joint as 𝐅_ID^i ∈ℝ^d, where d indicates the dimension of the identity representation. A CNN-based network f(·) is designed to obtain the identity representation. Mathematically, we have:

    𝐅_ID^i = f(𝐈_v00m^i).

The identity information guides and regularizes the diffusion process to not only focus on the target visual domain distribution but also maintain necessary information from the subjects. 

For the supervision of f to formulate identity information, it is proposed to conduct training in a contrastive learning manner. A triplet loss is introduced for this purpose as follows:

    ℒ_Triplet = ∑[  f(𝐈_v00m^i) - f(𝐈_v12m^i) _2^2 -  f(𝐈_v00m^i) - f(𝐈_v00m^k) _2^2 + α]_+,

where a triplet is formulated as (𝐈_v00m^i, 𝐈_v12m^i, 𝐈_v00m^k): 𝐈_v00m^i indicates the anchor of current subject's joint; 𝐈_v12m^i refers to a positive sample from the same subject's joint; 𝐈_v00m^k is a negative sample with i≠ k; and α is a margin enforcing between positive and negative pairs. 
Note that the number of triplets M used in the training process is defined in the experiment implementations. 

The identity information acts as a guide and regulator for the diffusion process, ensuring that it not only aligns with the target visual domain distribution but also preserves essential information pertaining to the subjects. To inject the identity into the diffusion process, 

a cross-attention mechanism is applied between the features from the UNet layers in ϵ_θ and 𝐅_ID. For the t^th diffusion step and j^th layer, the attention can be formulated as follows: 

    𝐀_j = Softmax(𝐐𝐊^⊺√(d))𝐕,
    
    𝐐 = ϕ_j(z_t)𝐌^Q_j , 𝐊 = 𝐅_ID^i𝐌^K_j , and 𝐕 = 𝐅_ID^i𝐌^V_j ,





where 𝐌^∘_j represents a projection matrix containing trainable weights with a projection dimension d, and ϕ_j(z_t) indicates the (flattened) intermediate representation of the UNet with j^th layer. The UNet can be formulated by: 

    ϕ_j+1(z_t) = ϕ_j(z_t) + 𝐀_j,


To this end, the IC-RDN can be learnt via the following scheme:

    ℒ_LDM-Identity := 𝔼_ε(x), ε∼𝒩(0,1), t[ ε - ε_θ(z_t, t, f(𝐈_v00m^i)) ^2_2 ].




 §.§ Generation-Guided KOA Progression Prediction
 

In addition to generating 12-month X-ray image 𝐈̂_v12m^i for prognosis, a singular progression severity grade is predicted as well based on the baseline visit X-ray image 𝐈_v00m^i and the generated 12-month visit X-ray image 𝐈̂_v12m^i. 
Specifically, a two-stream CNN backbone with two branches h_v00m(·) and h_v12m(·) is applied for the two images to produce the associated representations:

    𝐏_v00m^i=h_v00m(𝐈_v00m^i),  𝐏_v12m^i=h_v12m(𝐈̂_v12m^i).

The two representations are then concatenated, denoted by ‖, and used in a multi-layer perceptron (MLP) classifier g(·) to produce the predicted progression severity grade (ŷ_KL^i), which can be formulated as:

    ŷ_KL^i = g( 𝐏_v00m^i ‖𝐏_v12m^i)


A cross-entropy loss is leveraged in the training process to guide the prediction training process, which can be formulated as: 


    ℒ_CE = -∑_i=1^N y_KL^i logŷ_KL^i

where N indicates the total number of joints in the dataset. 



§ EXPERIMENTAL RESULTS AND DISCUSSION
 



 §.§ Dataset and Pre-Processing


The longitudinal X-ray images used in this study were sourced from the Osteoarthritis Initiative (OAI) <cit.>, which is a multi-centre and longitude study containing chronological medical imaging scans and clinical assessment records for the subjects up to 96 months. This public dataset enables researchers to explore the natural progression of osteoarthritis, identify risk factors, and evaluate biomarkers for grading the KOA severity and progression. In this study, the patients with both baseline and 12-month visits and available 12-month severity KL grades are selected. Due to some participants opting out of the follow-up study and missing recording of clinical severity grading outcome, not all have recordings for the 12-month visit image and severity KL grades. Consequently, 7,923 joints have both the KL grade available and radiographic recordings from both the baseline and 12-month visits. As the demographic information shown in Table <ref>, the amount of joints with changed severity grades among 12 months is much smaller than that of without changes, where only 6.15% joints show a changed severity grade. However, across these joints without the severity grade changes, the visual structural details do not remain the same, which also shows the importance of the proposed multifaceted progression predicting method. The dataset was randomly divided among subjects with a distribution of 70%, 10%, and 20% for training, validation, and testing, respectively. 



As the raw data from the OAI contains bilateral knee joints and includes almost all knee bones around the joint area, knee joint localisation methods are leveraged to extract the knee joint area individually <cit.>. After detecting the knee joint edge, a 224×224 region surrounding the middle of the joint was extracted from the raw data for modelling. The remaining transformation approaches for the joint images are followed <cit.>, including adjusting the brightness, contrast and saturation of the images. 




 §.§ Implementation Details



A pretrained ResNet 18 on ImageNet <cit.> is adopted as the backbone classification model for identity detection, which extracts the identity features from the baseline X-ray scans. Within Identity Prior's contrastive learning framework, we have designated the number of triplets M as 7,000 for training and 1,000 for testing. 
The Identity Prior training phase is fine-tuned using the Adam optimizer with a learning rate of 0.01, and the batch size for this phase is established at 128. 
For the training process of the diffusion-based generative module,  we set the number of steps to 1,000,000 across 400 epochs. 
The Adam optimizer is employed here as well, with a learning rate set at 1× 10^-4. The batch sizes for training and testing are 64 and 8, respectively. 

For the backbone models in the generation-guided KOA progression prediction module, various CNN methods are adopted, such as VGG 16, VGG 19, ResNet 18, ResNet 50, and DenseNet 121, for a comprehensive comparison <cit.>. Specifically, the backbone prediction models for both branches, corresponding to 𝐈_v00m and 𝐈̂_v12m, share the same architecture. In addition, the output features dimension from these backbone models is set to 128. Then the MLP classifier g(·) takes the input dimension of 256, which contains two linear layers, an activation layer, and a dropout layer. 
The training process is conducted with a batch size of 16. An SGD optimizer is utilized with the learning rate of 5× 10^-3 and a weight decaying factor of 5× 10^-3 for every 5 epochs.
The entire training regimen is executed using PyTorch 2.0.1 on an NVIDIA RTX A6000 with 48 GB of video memory.





 §.§ Quantitative Performance on Visual Prognosis


The evaluation of the generated future X-ray images encompasses both quantitative and qualitative assessments. Quantitatively, we engage in analysis through the conventional generation task metrics, inception score and a downstream task: the prediction of KOA progression severity grade. For using the inception score in this study, we use the Inception-v3 network trained on the OAI 12-month visit X-ray dataset to replace the weights pre-trained on the ImageNet <cit.> as the knee X-ray images are different from the natural images at both category and structural level. As shown in Table <ref>, the X-ray images generated by our proposed IC-RDN illustrate a similar IS with the X-ray images generated by the image-to-image diffusion model and outperform the X-ray images generated by Cycle-GAN <cit.> and other SOTA generative models <cit.> significantly. As for the similar IS between the proposed IC-RDN and the image-to-image diffusion model, the results show similar performance on image quality and diversity. However, 
the images generated by the I2I diffusion model show a minor contribution to predicting the KOA progression severity grades, and the challenge of identity inconsistency remains in the generated X-ray images from the I2I diffusion model. The detailed comparison for identity consistency is shown on Sec. <ref>. 

Although the inception score is able to show the quality and diversity of the generated future images, the downstream task of predicting the KOA progression severity grade can be more relevant to the value of how the generated future X-ray images contribute to exploring the KOA progression patterns. We have introduced the approach to predict KOA severity progression (Sec. <ref>), leveraging both the generated future X-ray image and the baseline X-ray image to determine a progression severity grade. 

In addition, as stated in Sec. <ref>, various CNN-based backbones are deployed in the generation-guided KOA progression prediction module. The comprehensive results among these backbone models are shown in Table. <ref>. 

Specifically, the predicted 12-month progression X-ray images generated by Cycle GAN, the image-to-image diffusion model, the recent SOTA generative methods, and the proposed IC-RDN are used in the generation-guided KOA progression grade prediction module separately. Accuracy is used as the evaluation method to measure the performance of predicting the progression grades. Among all backbone models used in this module, predicting the progression severity grades using X-ray images from baseline visit and generated by the proposed IC-RDN shows significantly outperformed results. In detail, the image-to-image diffusion model shows a limited contribution to providing the multifaceted KOA progression patterns. And the Cycle GAN illustrates the minor ability to enhance the performance of predicting the KOA progression severity grades. Therefore, the quantity evaluation demonstrates the added value of incorporating generated future X-ray images in enhancing the accuracy of KOA progression severity grade predictions.




To illustrate the performance of generated X-ray images contributing to predicting the KOA progression grades, the t-Distributed Stochastic Neighbor Embedding (t-SNE) technique, a sophisticated machine learning algorithm designed for the task of high-dimensional data visualization, is utilised in this study. In detail, to reduce the complex, multi-dimensional data into a two-dimensional space, t-SNE converts the high-dimensional distances between data points into conditional probabilities that represent similarities. Specifically, given a predicted X-ray image generated by the proposed IC-RDN, the trained VGG 19 network is used to produce a feature map corresponding to the KOA progression grade patterns. Then, these representations are visualised using t-SNE to show the related grades spatially. As shown in Fig. <ref>, the nodes with different colours indicate the corresponding grades. In Fig. <ref>.a, clear clusters for the KL grades of 0, 3, and 4 are located on the left and right sides. The green and yellow nodes referred to KL grades 1 and 2, show that the module for predicting the KOA progression severity grade has a minor ability to categorise these two classes, which is the challenge among different KOA progression prediction and severity grading methods <cit.>. However, in Fig. <ref>.b, the nodes for different grades are mixed and the cluster centres are hard to identify. 








 §.§ Qualitative Performance on Visual Prognosis



In terms of qualitative analysis, the comparison between X-ray images synthesized by the Cycle GAN and those produced by the proposed IC-RDN offers a compelling visual assessment, as depicted in Fig. <ref>. The comparative study highlights notable distortions in bone structures emanating from the Cycle GAN approach, particularly in the articulation of joint spaces. In stark contrast, our proposed IC-RDN exhibits a pronounced fidelity in preserving the integrity of key bone edges and the intricacy of inner textures, thereby underscoring the superior capability of IC-RDN in maintaining structural information.

Further elucidation is provided through an expanded corpus of X-ray images generated by the IC-RDN, which is shown in Fig. <ref>. This compilation serves not only to reinforce the initial observations regarding the preservation of critical structural details but also to demonstrate the consistent performance of the IC-RDN across a diverse set of examples. The consistent replication of complex anatomical features by the IC-RDN model is indicative of its sophisticated understanding of underlying bone structures, which is paramount for illustrating the multifaceted KOA progression patterns.

Moreover, this comparison elucidates the inherent limitations of traditional GAN-based approaches in contexts where the preservation of exact structural features is paramount. The propensity for distortion within Cycle GAN-generated images can significantly impact the clinical utility of such synthesized X-ray images, particularly in monitoring the KOA progression. In contrast, the proposed IC-RDN shows its adeptness at capturing and reproducing the nuanced details of bone structure suggesting its potential as a multifaceted KOA progression prediction outcome.







 §.§ Analysis on Identity Consistency



To further evaluate the efficacy of the identity prior module, we delve into a comprehensive comparison involving the X-ray images generated by the I2I diffusion model and the proposed IC-RDN. The visual examples presented in Fig. <ref> meticulously outline the distinctions between these methods alongside the actual 12-month visit X-ray images, where the key differences are highlighted by the bounding boxes. Initially, we observe the generated X-ray images devoid of identity persistency, which starkly contrasts with the subsequent imagery. 

A noteworthy observation is the preservation of identity features in the X-rays produced by our IC-RDN, as evidenced by the distinct texture and precision of bone edge structures. This preservation extends to intricate details within the knee joint, such as the orientation of the menisci and the nuanced protrusions along the bone edges, which are critical for maintaining identity persistency. 




 §.§ Ablation Study


To better evaluate the contribution of the proposed Identity Prior module, an ablation study is conducted by removing the Identity Prior and only the Diffusion backbone remaining. The results in Table <ref> demonstrate the effectiveness of the Identity Prior module with a superior longitude prediction accuracy among all selected classification methods. 





 §.§ Limitations


The major limitation of this study regards the generation quality. Directly using X-ray scans generated by IC-RDN for a KOA severity grading showcases a suboptimal result compared with using the baseline visit X-ray to predict the KOA progression grade directly. Though the predicted 12-month scan can enhance the KOA severity prediction with using the baseline visit jointly. In addition, computer hallucinations <cit.> are a crucial issue among large model fields. Particularly, the analysis regarding the disease-specific changes is insufficient. Therefore, careful consideration of applying the generative method to actual clinical usage is needed. 



§ CONCLUSION
 

In this study, a generative-based method, IC-RDN is proposed for KOA prognosis, which forecasts future X-ray scans for joints and singular progression KOA severity grades. This obtains multifaceted KOA progression patterns for understanding and explaining KOA prognosis.
Specifically, an identity prior module for the diffusion and a downstream generation-guided progression prediction module are introduced for regularising and guiding the diffusion to focus more on the clinical nuances related to the prognosis and producing a more precise KOA severity progression grading. 

Extensive experiments on a widely used public dataset, OAI, demonstrate the effectiveness of the proposed method.

In our future studies, the disentanglement of the representations related to the identity and KOA severity will be explored for fine-grained guidance in the X-ray image generation process. Meanwhile, the multimodal patterns among different medical imaging scans (X-ray, MRI, and CT) and patients' demographic information will be involved in producing a fine-grained KOA progression assessment outcome. 





§ ACKNOWLEDGEMENTS

This study was partially supported by Australian Research Council (ARC) grant DP210102674. The OAI <cit.> is a public-private partnership comprised of ﬁve contracts (N01-AR-2-2258; N01-AR2-2259; N01-AR-2- 2260; N01-AR-2-2261; N01-AR-2-2262) funded by the National Institutes of Health, a branch of the Department of Health and Human Services, and conducted by the OAI Study Investigators.






splncs04



 Paper Title: Multi-label Sewer Pipe Defect Recognition with Mask Attention Feature Enhancement and Label Correlation Learning.











a]Xin Zuo
a]Yu Sheng
b]Jifeng Shen
c]Yongwei Shancorrespondingauthor
[correspondingauthor]Corresponding author
yongwei.shan@okstate.edu


[a]School of Computer Science and Engineering, Jiangsu University of Science and Technology, Zhenjiang, 212003, China
[b] School of Electronic and Informatics Engineering, Jiangsu University, Zhenjiang, 212013, China
[c] School of Civil and Environmental Engineering, Oklahoma State University, Stillwater, OK 74074, USA







§ ABSTRACT 
The coexistence of multiple defect categories as well as the substantial class imbalance problem significantly impair the detection of sewer pipeline defects. To solve this problem, a multi-label pipe defect recognition method is proposed based on mask attention guided feature enhancement and label correlation learning. The proposed method can achieve current approximate state-of-the-art classification performance using just 1/16 of the Sewer-ML training dataset and exceeds the current best method by 11.87% in terms of F2 metric on the full dataset, while also proving the superiority of the model. The major contribution of this study is the development of a more efficient model for identifying and locating multiple defects in sewer pipe images for a more accurate sewer pipeline condition assessment. Moreover, by employing class activation maps, our method can accurately pinpoint multiple defect categories in the image which demonstrates a strong model interpretability. Our code is available at
https://github.com/shengyu27/MA-Q2Lhttps://github.com/shengyu27/MA-Q2L.



Sewer Pipe Defect Recognition and LocalizationMulti-Label LearningLocally Enhanced FeaturesLabel Correlation








§ INTRODUCTION


The degradation of sewer pipes over time, coupled with various external environmental factors, can lead to structural defects like pipe cracks, collapses, and fractures. These structural failures not only incur substantial economic losses for municipal utilities but also raise significant environmental concerns due to sewage overflows <cit.>. Consequently, timely maintenance and repairs are crucial. Regular inspections for sewer pipe condition assessment are fundamental to proactive asset management strategies. Through these inspections, maintenance and repairs can be scheduled in the early stages of defect occurrence, mitigating the risk of further deterioration.

In the early studies on this topic, sewer pipe inspection heavily relied on Closed-Circuit Television (CCTV) technology to capture video or image data. Subsequently, a trained operator manually annotated multiple defects by viewing the video footage or images. This method is typically inefficient, time-consuming, and labor-intensive <cit.>. However, with the development of automation and deep learning techniques in the field of computer vision, it is now feasible to utilize sensors or cameras to capture pictures or videos of sewer pipes. Deep learning algorithms are then employed to detect defects, and the detection results are presented to trained personnel for feedback <cit.>, <cit.>.

Given the inherent characteristics that multiple manually labeled defects may coexist in sewer pipe images <cit.>, multi-label classification techniques, capable of recognizing multiple defects simultaneously, are well-suited for such tasks. Moreover, in recent years, image multi-label classification techniques have garnered considerable success in the realm of deep learning. The method by Tao et al. <cit.> employs channel attention and spatial attention to gather the global contextual information of images and improve label correlation by constructing a model utilizing a graph attention network. However, this approach necessitates a two-stage procedure, which is time-consuming and difficult. Furthermore, channel attention may result in the loss of fine-grained position information, resulting in a failure to sufficiently collect position-related feature information. Another study <cit.> presents a multi-scale fusion-based end-to-end multi-scale hybrid vision transformer for modeling non-local spatial semantics. However, this multi-scale fusion is not suitable for images with multiple classes of defects coexisting, because pooling and down-sampling in the fusion operation result in the loss of the defect details and local information, rendering the model incapable of learning that information. To solve this issue, a mask attention method is proposed that focuses on the defective area information in order to increase the discriminative capacity of local spatial feature information and the classification performance of the model.

Due to the intricate nature of the sewer pipe environment, special attention needs to be directed towards the unique characteristics of the data resource. Sewer pipe defect images are derived from real-time video and are subject to post-processing. The defect categories in the images are diverse, with random features that are impossible to forecast manually. As a result, ensuring an equal distribution of defect categories in the final dataset is unachievable, resulting in a severe class imbalance issue known as the long-tail problem <cit.>. It is also illustrated in the literature <cit.> that there is indeed a long-tail problem with sewer pipe defect images, and it is also a continuing concern in the field of multi-label classification. To overcome this issue, an asymmetric loss function is used, and targeted category weight values are built to allow the model to alter parameters during backpropagation to improve learning ability for categories with fewer samples and more difficult classification tasks.


Furthermore, due to the variety of defects, one pipe image may be showing numerous problems at the same time. The coexistence phenomenon <cit.> can be clearly observed by analyzing Fig. 2(a) and calculating the occurrences of a specific defect category in conjunction with other specified defect categories in the Sewer-ML dataset. For example, when a crack defect occurs, it is common for a joint error or surface damage defect to also exist. Fig. 2(b) shows examples of images of defective sewer pipe. The upper image shows the following defects at the same time: cracks, breaks, and collapses (RB); surface damage (OB); displaced joint (FS); and connection with constructure changes (OK). Note that the 'Cracks' in Fig. 2(a) refer to RB defects. The lower picture shows defects such as cracks, breaks, and collapses (RB), displaced joint (FS), obstacle (FO), and branch pipe (GR). According to a qualitative study, pipe cracks are often produced by external forces or protracted corrosion owing to a wet environment and fluid transit <cit.>. As a result, the inner wall of a pipeline is frequently damaged, resulting in surface damage problems. Serious crack defects, if not maintained or repaired in a timely manner, can cause pipe fracture or even collapse, with the collapsed pipe material becoming an obstacle inside the pipe. To address the issue, self-attention computation is employed to help the model pay greater attention to the relevant information between labels to enhance label relevance and improve model performance.







To address these issues mentioned above, the authors propose a Mask Attention guided Q2L method (MA-Q2L) for multi-label sewer pipe defect recognition in complex sewer pipe environment. Firstly, a mask attention guided feature enhancement module is introduced, which leverages class activation map to obtain the local spatial discriminative information of different defects. The utilization of class activation map is preferred because they generate spatial discriminative information that is visualizable, providing further evidence that using class activation maps to explore local spatial feature information is effective. Secondly, the self-attention mechanism is incorporated to model the relationship between the labels, thereby improving label relevance. Since interaction learning between label embeddings and image characteristics may fail to capture the relationship and information between the labels, the self-attention computation allows the model to focus more on the correlated information between the labels, making the relationship between various labels clearer and more distinct. Finally, a specialized method based on the asymmetric loss function is given, which designs targeted category weight values. This allows the model to adjust parameters during the back propagation process, enhancing the learning capacity for categories with fewer samples and more challenging classification. The traditional asymmetric loss function struggles to effectively deal with the long-tail problem arising from a severe category imbalance in sewer pipe defect images while dynamically acquiring weight values to address potential inflexibility in the weight value acquisition process. Our proposed method has achieved the state-of-the-art performance on only 1/16 of the sewage pipe dataset Sewer-ML, and even 73.15% on the entire dataset, representing a significant improvement in model performance over prior models. In addition, as shown in Fig. 1, the method in this paper can localize the defect information in the image more accurately. 

The primary contributions of this paper are as follows:


	
  * An attention mask guided spatial feature enhancing method is proposed to improve the local feature discriminative ability.
	
  * The relevance of labels is improved through the self-attention mechanism of label embedding.
	
  * An asymmetric loss function with static and dynamic weight update technique is developed to relieve the class imbalance issue.
	
  * Our model not only achieves state-of-the-art (SOTA) performance on the multi-label classification task, but is also able to roughly localize pipe defects in the image, which demonstrates strong model interpretability.
	






§ RELATED WORK



 §.§ Automated sewer inspections

For decades, researchers and industry professionals have dedicated efforts to automatic sewer pipe defect recognition, aiming to achieve breakthroughs and develop efficient models for real-world applications. However, according to Haruum and Moeslund <cit.>, progress in this field has been slow, attributed to limitations in datasets and a scarcity of freely available modeling code. For the automated recognition and categorization of sewage pipe defects, previous studies predominantly utilized Closed-Circuit Television (CCTV) images <cit.>, <cit.>, <cit.> and other sensor-based approaches <cit.>, <cit.>, <cit.>. Furthermore, the lack of standardized datasets and assessment measures <cit.>, <cit.>, <cit.>, has impeded meaningful comparisons between various models, making the evaluation of their generalization effectiveness challenging <cit.>. 

After the release of the Sewer-ML dataset <cit.>, several researchers, such as Tao <cit.> and Haurum <cit.>, devised two-stage classification models. However, due to the intricacies and suboptimal performance of these two-stage models, this approach gradually lost favor. Haurum et al. <cit.> subsequently introduced an end-to-end network model, showcasing commendable performance. Nevertheless, this technique lacked in-depth exploration and utilization of the local spatial feature information within the original image. Concurrently, alternative methods <cit.> initiated multi-task classification, utilizing a graph neural network decoder to refine the prediction of each classification and identify issues and attributes related to sewer pipes. However, in terms of the model structure design, the defect classification received support primarily from information related to the other tasks, neglecting the discriminative local spatial feature within the image. Considering the specific characteristics of sewer pipes, additional approaches <cit.> aimed to diminish defect ambiguity by refining sewer pipe defects in the latent space, ultimately improving the performance of defect recognition and classification. 

While these technologies introduced novel perspectives, they often overlooked the inherent local spatial features of the original image. Our method extracts this information by generating the attention mask using the class activation map, which is then incorporated into the attention computation to enhance the learning capabilities of the model. This strategy facilitates a thorough exploration and utilization of the native local spatial feature present in the original image.

In addition to image-level defect recognition methods, there are also some pixel-level segmentation-based approaches. Mask R-CNN<cit.>, for instance, incorporates an additional branch network, employing ROIAlign to isolate segmentation targets and predict binary masks for each category independently. Sophisticated models, such as large model Segment Anything<cit.>, achieve remarkable segmentation performance on extensive datasets through prompt engineering. These models leverage a large number of manually annotated segmentation masks and achieve high-performance segmentation through loss constraints within the model. However, the Sewer-ML dataset utilized in this study lacks pixel-level segmentation masks. Therefore, the method proposed in this article relies on the model's efficient feature extraction capabilities to achieve effective results, producing rough segmentation renderings without relying on segmentation masks.



 §.§ Multi-label classification

Recent studies in the domain of multi-label classification have primarily concentrated on two key aspects to improve and optimize model performance. On one hand, efforts were made to optimize model performance using spatial features. Object detection methods, for instance, were initially utilized to pinpoint features, followed by subsequent feature extraction <cit.>, <cit.>. While this approach alleviated the complexity of feature extraction, acquiring prior information became more challenging, necessitating manual filtering and labeling of the labels, consuming substantial resources <cit.>. This proved unnecessary and undesirable for a classification model. Other researchers have explored the correlation between spatial features and labels <cit.>, <cit.>, <cit.>, linking them through model structure design. However, this approach required a substantial volume of labeled data to support model training, demanding an intricately designed model structure. In scenarios with limited data, the performance of the model would experience a significant downturn. Additionally, certain algorithms <cit.>, <cit.> aimed to classify images utilizing spatial location occlusion masks or multi-scale feature fusion. While the model could grasp finer-grained features using diverse location-based occlusion masks, in the presence of a more concentrated feature set, a considerable portion of features would be masked, leading to substantial loss of feature information. Moreover, in the process of multi-scale feature fusion, pooling and up-sampling contributed to the loss of information regarding local features.

On the other hand, researchers started delving into the correlation between labels. In multi-label classification dataset images, there might be a co-occurrence between labels. To address this phenomenon, linking these labels through graph neural networks or label embedding to enhance the correlation between labels <cit.>, <cit.>, <cit.> could enhance the classification accuracy of the model. However, this required considering the intricate connections between the labels of the dataset. For complex and specialized datasets, a more sophisticated design might be necessary, introducing more burdensome weights and computations.

Similarly, the authors embarked on a search in two aspects. Firstly, a class activation map is utilized to generate the attention mask, enhancing the capacity of the model to learn diverse spatial features related to defects. This is akin to obtaining prior information from the class activation map, which is substantially less expensive than using object detection method. Secondly, for label relevance, label embedding is employed for self-attention computation, enhancing the learning capacity of the model for label-to-label interactions.



 §.§ Loss function and long-tail problem

The cross-entropy loss function <cit.>, <cit.> was commonly utilized in multi-label classification, contributing to a significant performance gap across labels due to the inherent long-tail issue in multi-label datasets. Subsequently, focal loss <cit.> was inextricably linked to multi-label categorization. Focal loss augmented cross-entropy with configurable parameters to alter the balance between positive and negative samples, allowing the model to focus more on challenging-to-classify negative samples for dataset balance and enhanced performance. However, it controlled both positive and negative samples with the same parameter, increasing the contribution of negative samples while decreasing the contribution of positive ones.

The ASL(Asymmetric Loss) <cit.> decoupled the parameters of focal loss, adjusting the learning capacity of the model for positive and negative samples separately, rendering it more appropriate for datasets with the long-tail issue. Some researchers have already employed ASL for multi-label classification <cit.>, <cit.>, <cit.> with promising results. However, the ASL parameters were statically specified and might not be adequately fitted to a specific dataset. 

Considering the presence of the long-tail problem in sewer pipe defect data, the authors employ ASL. Simultaneously, the weight values are elevated for bottleneck categories characterized by fewer samples and greater classification difficulty. During backpropagation, the model parameters are adjusted to facilitate enhanced learning for these bottleneck categories. This method can better deal with the problem while retaining the original control for the contribution of positive and negative samples, thereby improving the overall performance of the model. This approach straightforwardly and effectively addresses the issue while preserving the original control over the contribution of positive and negative samples, leading to an overall improvement in model performance.




§ THE PROPOSED METHOD

 

In this section, firstly, the overall structure of a model is given (Fig. 3), which is inspired by Q2L <cit.>. Meanwhile, the authors chose Q2L as our baseline since it has commendable performance on public multi-label classification datasets. Our model uses Resnet101 as the backbone and then employs the classic Transformer structure, the encoder-decoder structure, to combine label embedding for interactive learning. An attention mask, derived from the backbone using a class activation map, is then integrated into the decoder. The label embedding information undergoes linear mapping to achieve multi-label image classification. In contrast to the Q2L baseline, our model substitutes the original backbone with an improved backbone, integrates a custom-designed attention mask into the encoder, and adopts an enhanced asymmetric loss function in lieu of the original loss function. Subsequent sections will delve into the detailed explanation of each module.




 §.§ Problem formulation


Given an input image I∈ℝ^3 × h × w, the authors aim to design a model for predicting a multi-label vector O∈{ 0,1 }^1 × N representing the defect categories present in the image. In this binary encoding, 1 indicates the presence of the corresponding defect category, while 0 signifies its absence. The model comprises three functions, F_backbone, F_encoder and F_decoder. The function F_backbone is employed to extract features from the raw input image to obtain the feature representation X∈ℝ^C × H × W. The function F_encoder processes these features further and captures the information between image features to get the encoded feature X_e∈ℝ^C × HW.  F_decoder interacts with the label embedding which directly passes the label through the embedding operation and extracts its weight value, denoted as LE, to learn from the extracted features X_e and ultimately produces the predicted result O∈ℝ^1 × N. The specific definitions of these functions can be denoted as:


    X = F_backbone(I) 
    X_e = F_encoder(X) 
    O = F_decoder( X_e,LE)

where h and w are the height and width of the input image I, respectively, H, W and C are the height, width, and channel number of the feature map. N is the total number of defect categories.





 §.§ Encoder


The authors adopt a convolutional structure as the backbone, which can model and learn local information. In order to capture the global context semantic feature information, the features are fed into the encoder structure and further processed to get high-level semantic features. The encoder includes a multi-head self-attention mechanism, enabling the learning of multiple distinct feature representations. Each attention head focuses on different local information within the input sequence or features, capturing richer semantic information. Specifically, h denotes the number of heads, and different projection matrices are used to generate different tensors: . { Q ._i}_i = 1^h, . { K ._i}_i = 1^h, . { V ._i}_i = 1^h. Multi-head self-attention can be formulated in Eq. (4)-(6).

    Attention( Q,K,V) = softmax( QK^T / √(d_k))V
    
    	MultiHead( Q^',K^',V^') = Concat( head_1,…,head_h)W
    head_i = Attention( Q_i,K_i,V_i)

where Q^' is the cascade of . { Q ._i}_i = 1^h, K^' and V^' are also the same settings, and W is the linear projection matrix. The whole encoder structure can be represented as:

    F_0 = MultiHead( TE(X),TE(X),TE(X) ) 
    F_1 = Norm( X + dropout( F_0) ) 
    F_enc = Norm( F ._1 + dropout( FFN( F_1) ))

where TE stands for token embedding, Norm and dropout stand for normalization operations to prevent overfitting, FFN stands for feed-forward network, and F_enc stands for the final feature representation of the encoder structure, where the positional information needs to be added in the attention computation, which is not represented in Eq. 7 for clarity.




 §.§ Attention mask


In addition to entering the encoder structure, another path for features from the backbone is to generate the attention mask through class activation map. Let’s first introduce the process of generating the class activation map: the feature map X∈ℝ^C × H × W is obtained from backbone. Then, after global average pooling, the height and width dimensions of the feature map are averaged to yield a global feature vector. Next, the vector is linearly mapped to the same dimensions as the number of defect categories to obtain the weights which represents local spatial feature information about the defects in the corresponding image. Subsequently, the weights are fused with the original feature map to generate the class activation map, serving as the attention mask. 
Specifically, for each pixel in the image, the model assigns different values according to different defect categories, indicating whether the model pays attention to the pixel or not. Larger values imply greater model attention, suggesting a higher likelihood of the presence of the defect category at that location. Conversely, smaller values mean that the model pays less attention to the pixel, considering it less relevant information. The process of generating the weights is depicted in Eq. 10:

    A_0 =  L_W( GAP(X) )

where X represents the feature map obtained from backbone, GAP is global average pooling, L is linear mapping, and W represents the weights of the linear mapping layer. A_0 is the classification probability result, which is not needed. The authors only need the weights W which are fused with the feature map X to generate the attention mask, as shown in Eq. 11:

    A_1 =  X∗W

where W∈ℝ^C × N, A_1∈ℝ^HW × N is the attention mask.
The attention mask is integrated into the subsequent computation of the attention mechanism, thereby augmenting the learning capability of the model. Importantly, this process is akin to acquiring prior information from the class activation map, contributing to enhanced performance in the defect recognition classification task.



 §.§ Decoder


To capture information from the labels, the decoder structure is followed by the encoder, enabling interactive learning between the label embeddings and the features from the encoder to enhance model performance. The label embedding setting of the model is adopted from Q2L to obtain the query and its iterative update method. In the cross-attention of decoder, the key and value come from the encoder F_enc, while the query originates from the label embedding LE, representing the label information. Cross-attention enables dynamic interaction between label embeddings and image features, effectively modeling the relationship between features and labels. Ultimately, the query learns the presence of specific categories in the image, facilitating binary classification decisions.

At the same time, the attention mask is integrated into cross-attention. The information in the attention mask is added to the results of query and key computations, prompting the model to pay more attention to the meaningful defect local spatial discriminative information region during interactive learning. This approach enhances the accuracy of defect detection in the image, contributing to overall model performance improvement. Moreover, relying solely on the interactive learning between image features and label embeddings may not capture the intricate label relationships adequately. To address this, self-attention computation is employed, making connections between different labels more explicit and distinct. Therefore, this enhancement learning for label relevance by performing self-attention computation is shown in Fig. 4, which is structured as:


    CA_i( Q_i,K,V,A_1) = Softmax( Q_iK^T/√(d_k) + A_1)V
    Q_t = Norm( Q_i + dropout( CA ._i( Q_i,K,V,A_1))) 
    . Q_t + 1 = Norm( Q_t + dropout( MultiHead( Q . . ._t,Q_t,Q_t))) 
    . Q_i + 1 = Norm( Q_t + 1 + dropout( FFN( Q . . ._t + 1)))

where, Q_i represents the query in the i-th decoder. The initial query, Q_1, is obtained from label embedding LE. K represents the key, and V represents the value, both come from the encoder F_enc. A_1 represents the attention mask. CA_i represents the output of the i-th cross-attention calculation. Q_i + 1 represents the output of the whole decoder, which also serves as the input of the next query of decoder. Q_t and Q_t+1 represent the temporary variables required in the decoder after self-attention computation in Eq. (13)-(15). Among them, the attention computation needs to incorporate the position information, which is not represented in Eq.14.





 §.§ Static loss weight

Our model employs a targeted category weighting technique based on an asymmetric loss function to effectively deal with the long-tail problem in sewer pipe defect data. In multi-label classification, it is customary to simplify the problem into a series of binary classification tasks. Given N labels, the basic network produces one logit value per label, Z_n. Each logit is independently activated by a sigmoid, and y_n serviced as the ground truth for the category n. The total classification loss, L_total, can be obtained by aggregating the binary classification losses from the N labels. As shown in Eq.16.

    L_total = ∑_n = 1^NL( sigmoid( Z_n),y_n)

The general form of the bisection loss L for each label is shown in Eq.17:

    L = - yL_+ - (1 - y)L_-
    { L_+ = (1 - p)^γlog(p)
    L_- = p^γlog(1 - p).

where y is the ground truth label (category index n is omitted for convenience), and L_+ and L_- denote the positive and negative loss components, respectively. γ is a parameter that controls the extent of learning of the model for positive and negative samples, and p = sigmoid( Z_n). Focal loss is introduced to address the long-tail problem of the dataset, and controlling γ allows the model to focus on learning the categories that are challenging to categorize, thereby amplifying the contribution of negative samples and diminishing the contribution of positive samples.

ASL decouples γ in the original focal loss into two parameters, γ_+ and γ_-, as a means of controlling the learning degree of the model for positive and negative samples, respectively. ASL reduces the contribution of negative samples to the loss by thresholding m, solving the problem of insufficient contribution to positive samples presented in the focal loss. Experimental evidence <cit.> has demonstrated that setting  γ_+ = 0 and γ_- to a static value of 2 yields the best result compared to the dynamic value.

On top of ASL, the authors introduce multiplicative weights α to the binary loss function L for each label. This adjustment increases the loss share of the bottleneck categories when calculating the overall loss, channeling the learning of the model focus towards these specific categories. This concept parallels the design of ASL for γ_-, which tunes the contribution of negative samples to the overall loss, accentuating challenging negative samples. Similarly, our α aims to modulate the learning capacity of the model for the bottleneck categories, regulating their contribution to the overall loss. For easily recognizable categories, α remains at its original weight of 1, while for harder-to-recognize categories, α is set to 2.

By introducing the weight updating strategy of α, the authors enhance the learning for the bottleneck categories. This approach addresses the loss between different categories in a more balanced manner, improving the ability of the model to recognize bottleneck categories and further enhancing the performance of the sewer pipe defects classification task. The specifics are as follows:

    L_total = ∑_n = 1^Nα L( sigmoid( Z_n),y_n)
    { α = 1,  n ∉ B
    α = 2,  n ∈ B. 
    
    	B = {RB,IS,FO,OS}

where n represents a certain category and B represents the set of bottleneck categories, which contains the four defect categories, and the selection of this set is described in experimental section 4.3.



 §.§ Dynamic loss weight

In refining our loss function, the hyperparameters are typically tuned through manual experimentations, which is a laborious and time-consuming process. Specifically, determining the suitable α weight poses a challenge. To address this, a dynamic and learnable method for the values of α is introduced. The authors gauge the difficulty of recognizing and classifying each category by computing its mean average precision (mAP). This assessment relies on the variance between the mAP of each category and the overall mean mAP, scaled to a single-digit value, which is then employed as the α value. Specifically:

    Mean_mAP = 1/N∑_n = 1^NmAP_n
    ΔmAP_n = Mean_mAP - mAP_n
    α_n = ΔmAP_n/10

where mAP_n represents the mAP value of n category. Mean_mAP represents the mean value of all categories. ΔmAP_n represents the difference between the mAP of n category and the mean value of the mAP of all categories, and a_n represents the weight value to be added to n category. The algorithm, using difference calculation, can be seen as evaluating the gap between each category and the mean value, reflecting the overall performance of each category under the training of the model. After this, the authors sort a_n in descending order, taking the first m corresponding weight values set to the weighted values of the corresponding categories, and setting the other values to 1, allowing the model to weight only the first m categories that were not learned well enough, while keeping the focus on the other categories unchanged, specifically:

    α_n = {α_n, n ≤ m
    1,n > m.

where m represents the number of top values to be selected for category weights after descending sorting of weights. 

Considering that the evaluation metrics are F1_Normal and F2_CIW, experimental analysis revels that the sequence and magnitude of mAP values for each category align with those of F1_Normal and F2_CIW. Hence, opting for the mean of mAP values is consistent with choosing the mean of either F1_Normal or F2_CIW values. The selection of either method yields an equivalent effect.




§ EXPERIMENT

In the experimental section, the authors commence by detailing the experimental setup, followed by outlining the process for selecting the one-sixteenth sub-dataset of Sewer-ML. Subsequently, the authors elucidate the experiments conducted to identify the bottleneck categories, concluding with the presentation of results where our model is compared to the baseline as well as to other models.



 §.§ Experiment setup



  §.§.§ Experiment details

In constructing our model, the authors adhere to the settings in the Q2L model. The backbone of our model is Resnet101, pre-trained on ImageNet, while the encoder and decoder utilize the official Transformer structure modules <cit.>. All images are resized to 448 × 448 as the input to the model. The feature dimension after backbone extraction is H × W × C = 14 × 14 × 2048. For training, the authors employ the Adam optimizer <cit.> with a learning rate of 1e-5, a weight decay of 1e-2, a batch size of 16, and 40 epochs of iterative training. Additionally, the authors apply an exponential moving average (EMA) to model parameters with a decay of 0.9997. Our code will be made publicly available, and more detailed settings can be found in the code repository.



  §.§.§ Evaluation metrics



Following the evaluation metrics in <cit.>, F1_Normal and F2_CIW are used as metrics to evaluate the model performance. In order to bring in domain knowledge, they both follow F_β:


    F_β = (1 + β^2)Precision × Recall/( β^2× Precision + Recall ) 
    
    	Precision =  TP/(TP + FP) 
    
    	Recall =  TP/(TP + FN)


where, β is the weight of recall and F_β considers recall to be β times more important than accuracy. TP and FN represent the number of correctly or incorrectly detected defective samples, while FP denotes the number of non-defective samples incorrectly classified as defective. In the sewer pipe defect classification task, the metric of normal image is denoted as F1_Normal, i.e., β = 1. For defective images, the class importance weight (CIW) needs to be incorporated into the calculation, as illustrated in Table 1. Moreover, because FN has a greater economic impact than FP <cit.>, β = 2 needs to be set.

    F2_CIW = ( ∑_n = 1^NF2_n×CIW_n) / ∑_n = 1^NCIW_n

where F2_n and CIW_n are the F2 and CIW values of n categories, respectively, and N is the total number of categories.






 §.§ Split of Sewer-ML dataset

Considering the extensive data in the Sewer-ML dataset, it was divided for experimental convenience. The complete Sewer-ML dataset, consisting of 75,618 video crops related to sewer pipelines with expert annotations <cit.>, has a continuous distribution. Images within the same sequence originate from a single video and exhibit a high degree of similarity. In such scenarios, a straightforward image selection strategy, combined with an appropriate dataset size percentage, is essential for facilitating model training and assessing performance improvements. Therefore, a sampling approach was employed, selecting every tenth image to create a smaller dataset.

Given that the training dataset comprises 1,040,129 images and the validation set contains 130,046 images, the authors opted for a one-sixteenth segmentation to balance computational efficiency and dataset sparsity. Further segmentation, such as one-thirty-second, would exacerbate the sparsity issue, especially for defective categories such as IS, FO, PB, OS, and OP, rendering it insufficient for effective model training, as shown in Table 2. Consequently, the chosen one-sixteenth sub-dataset includes 65,008 training images, striking a balance between training time and data adequacy. The validation set is obtained using the same operation, as shown in Table 3.





In the sub-dataset, there are only 30,916 normal defect-free images, accounting for 47.56% of all images, compared to 53.15% in the entire dataset. So, the evaluation metrics for normal images, F1_Normal, are lower than the results obtained by others when using the sub-dataset in the experimental section. To further compare the distribution of defects in the sub-dataset and the entire dataset, the authors normalize the number of samples for each defect category using the minimum-maximum method. Specifically:

    Value_n = ( Value ._n_ o - Min(Value))/( Max( Value) - Min(Value) )

where, Value is the set of sample numbers of all categories, Value_n_ o and Value_n represents the sample number of n category before normalization and after normalization, respectively. Max represents to get the maximum value in the set, Min represents to get the minimum value in the set.

Furthermore, as depicted in Fig. 5, the defect distribution in the sub-dataset closely mirrors that of the entire dataset, suggesting that training the model on the sub-dataset is representative of the full dataset. Our focus remains on the training and validation sets since the test set is not publicly accessible.

 




 §.§ Selection of bottleneck categories

 

 


To effectively address the long-tail problem in sewer pipe defect images, the authors assign higher weight on the bottleneck categories that pose challenges in classification and have a relatively small number of defects. Referring to the best prior results <cit.> on the Sewer-ML dataset, RB, IS, and FO are initially identified as the bottleneck categories, as illustrated in Fig. 7. The validation of this identification is evident when training the baseline model <cit.> on the sub-dataset, as shown in Fig. 6. The authors calculate and find that the average value of F2_CIW for 17 defect categories is 70.06%. Consequently, the authors focus on the categories with a F2_CIW value below the average to determine the bottleneck categories.

Aside from the three defect categories with the lowest performance (RB, IS, and FO), there are secondary defect categories (DE, IN, RO, AF, and OS) falling in the 60%-70% range. These are considered as potential bottleneck categories. Since our F2_CIW is computed with the greatest economic impact, the influence of class importance weight on the results must be considered. Among these possible bottleneck categories, the OS has the highest class importance weight, reaching 0.9009. This implies that the detection and classification of the OS defect category significantly impact the performance of the model and indicates a severe economic impact, warranting special attention. Consequently, the authors define RB, IS, FO, and OS as the four bottleneck categories. 




 §.§ Comparison with state-of-the-art


When training and validating on the sub-dataset, our F2 metrics reach SOTA, achieving 67.03%, as shown in Table 4. In addition, considering the potential impact of the incompleteness of the validation set on the final results, the authors also conduct validation on the complete dataset. As the amount of data increases, the F2_CIW metric decreases but is only 1.0 percentage points worse than the current best result of 63.38%. This gap is deemed acceptable, given the training dataset reduced by a factor of 16. The lower F1_Normal is attributed to using the sub-dataset for training, where normal images account for only 47.56% of the total, compared to other models trained on the entire dataset where normal images account for 53.15%. When trained on the whole dataset, F1_Normal improves, outperforming all prior results, indicating that our model is also excellent at recognizing normal images. Notably, the data in Table 4 are all from the original paper, and our model is configured as follows: attention mask, label relevance, and α = 2.

It's worth noting that the baseline surpasses the current best result when using only the sub-dataset, showcasing the efficacy of our chosen baseline. Furthermore, our model outperforms the baseline by 3.63 percentage points on the sub-dataset and 1.49 percentage points on the full validation dataset. While the performance of baseline improves with the increase in data on the entire dataset, our model demonstrates greater efficiency, achieving a 1.9 percentage point improvement to 73.15%, surpassing the prior best result in previous research by 9.77 percentage points.

On the test set, our model continues to excel, surpassing the F2 score by 11.41 percentage points. This performance is corroborated by Haurum and Moeslund <cit.>, demonstrating the excellent performance of our model in recognizing and classifying defects in sewer pipe images. It is worth noting that the results on the test set were not provided in the paper by the other authors, so the authors have no way to compare them.



Furthermore, in comparison to the best model result, MHSViT <cit.>, previously presented for each category F2_CIW, as shown in Fig. 7, our method improves performance across the categories. The most substantial improvement is observed in the bottleneck categories RB and IS, which are the most challenging to recognize, with an improvement of over 20 percentage points on average. This noteworthy and unprecedented improvement underscores that our approach has made significant strides in addressing the issue of category imbalance.





§ ABLATION EXPERIMENT

In this section, sub-datasets, including the validation set, are employed for experimental convenience. Firstly, the weights are determined by ablation experiments based on the selection of the bottleneck categories in section 4.3. Subsequently, the correctness of the bottleneck category selection is experimentally demonstrated. Next, the authors present the different backbone options based on baseline and discuss the specific performance of each component of our improved structure. Finally, experimental details about dynamic weight values are presented.




 §.§ Static value weight

 

Based on our theoretical analysis in section 4.3, the authors select RB, IS, FO, and OS as our bottleneck categories. By enumerating several values, it is observed that the weight value α = 2, F2_CIW, produces the best results. This demonstrates that increasing the weight value effectively allows the model to learn the bottleneck categories in a targeted manner, thereby enhancing the overall performance of the model, as shown in Fig. 8. When α = 1, without increasing the weights, the model is not allowed to target learning. It is noteworthy that when α exceeds 2, F2_CIW begins to gradually decrease, suggesting that imposing too large weight value will cause the model to focus too much on the bottleneck categories and neglect the learning of other defective categories, which is undesirable. Therefore, the authors decided to set the weight value α to 2, a static value, as the final model parameter, allowing us to target the bottleneck categories while still retaining sufficient learning for other defective categories to achieve the best performance.






 §.§ Verification of bottleneck categories


In order to verify our theory about the bottleneck categories, the authors conduct weighted experiments on different combinations of categories. Initially, the three categories are validated which are the most difficult to categorize in prior study results: RB, IS, and FO. However, the results were not satisfactory, as shown in Table 4. Subsequently, according to our theory in section 4.3, the authors add DE, RO, IN, AF and OS to the set of bottleneck categories respectively. The results support our theory: since OS has the largest proportion in the CIW, it has the greatest impact on the results when calculating the F2_CIW, leading to improved model performance. These experimental results demonstrate that our selection of bottleneck categories and weighting strategy is acceptable and proper, capable of achieving considerable performance improvement on sewer pipe defect data. By focusing on the bottleneck categories for enhanced learning, our model effectively addresses the category imbalance problem of the dataset and achieves excellent classification performance.






 §.§ Backbone

The authors select the recent better performing multi-label classification models <cit.>, <cit.>, <cit.>, and find that Q2L-resnet101 achieved optimal performance, even when the other model has larger input size, as shown in Table 5. Therefore, Q2L is selected as our baseline. To explore the effect of different backbones on the model performance, the authors adopt <cit.>, <cit.>, <cit.>, <cit.> as the backbone and follow the configuration in the corresponding paper. It is worth noting that due to the different backbone settings, the model also has different input image sizes. Larger image input can provide more image information, but even with the model using Tresnet-L as the backbone, the performance with an input image size of 448 × 448 was not as good as expected. On the other hand, for smaller sizes of the input image, the performance difference with the model employing resnet101 as the backbone is significant, and the size of the input image is not a key factor in bridging this performance gap. In addition, from the comparison of the number of parameters, the model with resnet101 has the lowest number. Considering the above factors, the authors  finally choose the Q2L model with resnet101 and with an input size of 448 × 448 as our baseline, as it achieved better performance results while keeping the number of parameters relatively low.







 §.§ Effect of each module

The authors verify the impact of each part of the model on the performance by adding modules or structures independently step by step. In Table 6, the original baseline achieved 63.4% performance, which is already the current optimal result. This indicates that the baseline the authors chose is very suitable for the task of sewage pipe defect identification and classification. 

With the addition of the attention mask, the performance is improved by 2.36 percentage points, which is the most significant improvement in our improved module. This result proves that the attention mask generated through the class activation map is very effective in mining and utilizing the defective local spatial discriminative information. And, incorporating self-attention computation into the decoder improves the performance by 1.54 percentage points. Due to the coexistence of multiple classes of defects in the image, the model can better capture the co-occurrence relationship between different labels for more accurate classification through self-attention computation. Furthermore, the enhanced learning of the bottleneck categories results in a 1.82 percentage point improvement in the overall performance of the model, further addressing the category imbalance of the data. Finally, integrating all three aspects of our improvement together, the overall performance reaches 67.03%, achieving a 3.61 percentage point improvement compared to the baseline. This fully proves that our improvement is effective. Previous research has found that specific defects in sewer pipe images are especially difficult to recognize, making the images challenging to apply in real-world applications. Our enhancement has significantly increased performance and advanced the modeling application process.






 §.§ Dynamic value weight


 
In order to explore the impact of different values of α on the model performance, the authors designed a dynamic way of determining α. Following section 3.6, the authors experimented with different values of m to allow the model to dynamically select the set of bottleneck categories that need to be augmented for learning during the training process. As shown in Fig. 9, our model achieves the best performance when m = 4, demonstrating that using four classes as the bottleneck categories is the best combination result. This also verifies that our analysis of the bottleneck categories and the number of selections in section 4.3 is correct. However, the outcomes of weighing using dynamically generated weight values are inferior to those produced using static values. The main reason for this might be that the weight value α is calculated based on the mAP of each training iteration, and the weight value keeps changing. At the beginning of the training, it is possible that the bottleneck categories are not weighted, but rather other categories with high confidence originally, which may cause the model to ignore the bottleneck categories even more. As the number of iterations increases, the model gradually focuses on the bottleneck categories. When training converges, it is likely that the model has not sufficiently learned the defective features of the bottleneck categories, leading to a lower overall accuracy. Therefore, the authors finally decided to use static values instead of dynamic values to ensure that the model focuses on the bottleneck categories throughout the training process and maintains consistent weight assignments in all situations, which can improve the overall performance.







 §.§ Different input size


 

 


 




To explore the relationship between different resolutions and accuracy, experiments were conducted using various input scales to train the model. Loss comparisons and accuracy changes for training at different resolutions were provided. For experimental convenience, experiments were conducted on sub-datasets. Overfitting in the model was observed based on the training and validation losses, in Fig. 10 and Fig. 11. To address this issue, the same strategy as Q2L<cit.> was employed, which involved early stopping. Given the large number of images in the dataset, the model tends to learn more information per epoch, resulting in rapid convergence. Early stopping is typically triggered after training for approximately 20 epochs.
Images of different resolutions contained varying amounts of information. The larger the resolution, the more information the model could potentially learn. As shown in Fig. 10, the training losses for four different scales were initially sorted according to scale size. However, it was important to note that a larger scale also meant the model was more likely to capture noise, leading to increased computational complexity. As the training epoch progressed, it became evident that the loss values tended to increase with larger scales, as depicted in Figure 10. This phenomenon could be attributed to increased noise interference, necessitating longer training iterations.
However, despite encountering noise at larger scales, significant performance improvements were observed in Fig. 11 and Fig. 12. These improvements manifested as lower losses on the validation set and higher F2_CIW indices.
Additionally, considering that the training loss for the 448 × 448 scale was lower than that for the 576 × 576 scale, indicating relative stability, and the latter incurred nearly double the computational cost, the modest 0.14% improvement in accuracy did not justify the increased computational burden. Therefore, the model finally adopted an input size of 448 × 448.







§ VISUALIZATION AND DEFECT LOCALIZATION


In this section, the visualization analysis is first performed for the bottleneck categories. Subsequently, eight defective categories are selected for visualization to further underscore the effectiveness and the interpretability of our approach in mining local spatial features.



 §.§ Visualization of bottleneck categories


 

Our model offers sewer pipe repairers with a visual representation of a rough heatmap of image defects. The matrix within this heatmap is adjusted by applying a threshold to the weight values in the class activation map, selecting only values surpassing the threshold, and setting the rest to zero. Then, non-zero values are binarized to 1, and object contours are extracted from the map using OpenCV library methods. The resulting matrix is further filtered based on area judgments.

The heatmap results for displaying the four bottleneck categories are presented in Fig. 13. The model effectively identifies distinct defect categories even when they co-exist with other defects and are located at uncertain spatial positions in the image. However, some issues were identified through the rough defect localization map. Since the images are selected frame by frame from the video, the limited illumination area in the video can lead to the deepest part of the pipe not being well-lit. This may cause the model to mistakenly infer the presence of defects, especially for OS defect, located on the inner wall of the pipe closer to the darker area. This scenario makes it challenging for the model to focus its attention, leading to distractions and false positives, as observed in Fig. 13(d).

Although our model can effectively and roughly localize the area showing the defects, challenges persist due to the limitation of the illuminated area by captured light. Beyond our work, addressing this peculiarity may further enhance the recognition of defective images in sewer pipes, and this will be a focus of our future work.



 §.§ Defects Localization

Our model roughly localizes the areas of defect categories by means of heat maps. The authors give some examples of common defect combinations to show the ability of our model to roughly localize defects regions, as shown in Fig. 14.

 




§ CONCLUSION

In order to deeply mine and utilize the local spatial feature information of defects, the authors adopt the class activation map to generate the local spatial discriminative information of defects as the attention mask and participate in the attention computation to achieve feature enhancement. Also, for the coexistence of multi-class defects and serious class imbalance problems in the sewer pipe defective images, the authors adopt the self-attention mechanism of label embedding to learn the relationship between labels and enhance the label relevance. In addition, the authors use a weight updating strategy based on an asymmetric loss function for bottleneck categories enhancement learning to further address the category imbalance problem in the data. Since the dynamic value approach fails to show as much superiority as the static value approach, the authors ultimately choose to use the static-value weight update strategy. Our model achieves results approximating SOTA on just one-sixteenth of the Sewer-ML dataset, and outperforms the current best method F2 performance by 11.87% on the full dataset. In addition, our model can provide rough heatmaps for defect localization to workers repairing pipes, and the authors hope that our work will advance the field of defect recognition in sewer pipes. Our future work may further investigate this area from the perspective of the wrong region of interest due to insufficient light deep in the pipe.



§ DATA AVAILABILITY STATEMENT

Some or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request. 




§ ACKNOWLEDGEMENT

This work was supported in part by NSF of China under Grant No. 61903164 and in part by NSF of Jiangsu Province in China under Grants BK20191427.













 Paper Title: Domain Shift Analysis in Chest Radiographs Classification in a Veterans Healthcare Administration Population.








    Curvature-Adapted Hypersurfaces of 2-type

in non-flat quaternionic Space Forms
    Ivko Dimitrić
    
===============================================================================


    ^1 Advanced Computing for Health Sciences, Computational Sciences and Engineering Division,
 Oak Ridge National Laboratory 

     ^2 IT Services Division, Oak Ridge National Laboratory 

     ^3 Veteran Affair's Association 

     ^4 Los Alamos National Laboratory 

     ^5 Yale University 

     ^6 Stanford University 

     ^7 Department of Biomedical Informatics, Vanderbilt University Medical Center 

     ^* Corresponding Author Email: chandrashekm@ornl.gov
     

§ ABSTRACT 

Objectives: This study aims to assess the impact of domain shift on chest X-ray classification accuracy and to analyze the influence of ground truth label quality and demographic factors such as age group, sex, and study year.

Materials and Methods: We used a DenseNet121 model pretrained MIMIC-CXR dataset for deep learning-based multilabel classification using ground truth labels from radiology reports extracted using the CheXpert and CheXbert Labeler. We compared the performance of the 14 chest X-ray labels on the MIMIC-CXR and Veterans Healthcare Administration chest X-ray dataset (VA-CXR). The VA-CXR dataset comprises over 259k chest X-ray images spanning between the years 2010 and 2022. 

Results:
The validation of ground truth and the assessment of multi-label classification performance across various NLP extraction tools revealed that the VA-CXR dataset exhibited lower disagreement rates than the MIMIC-CXR datasets. Additionally, there were notable differences in AUC scores between models utilizing CheXpert and CheXbert. When evaluating multi-label classification performance across different datasets, minimal domain shift was observed in unseen datasets, except for the label "Enlarged Cardiomediastinum." The study year's subgroup analyses exhibited the most significant variations in multi-label classification model performance. These findings underscore the importance of considering domain shift in chest X-ray classification tasks, particularly concerning study years.

Conclusion:
Our study reveals the significant impact of domain shift and demographic factors on chest X-ray classification, emphasizing the need for improved transfer learning and equitable model development. Addressing these challenges is crucial for advancing medical imaging and enhancing patient care.





 Notice: Office of Science of the U.S. Department of Energy. This manuscript has been authored by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The US government retains and the publisher, by accepting the article for publication, acknowledges that the US government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for US government purposes. DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan). 





§ INTRODUCTION



Chest radiography is the first-line imaging test for respiratory disease and some forms of cardiac disease. Chest X-ray abnormalities detection has been automated by recent advanced artificial intelligence techniques. Accurate chest X-ray classification has played an important role in many biomedical applications to accelerate the diagnosis and treatment of conditions like pneumonia, heart failure, rib trauma, pulmonary fibrosis, etc. Though there has been an increase in the integration of machine learning models to enhance diagnostic promise, the efficacy of these models hinges on the availability and quality of training data. 

Recent years have seen a wave of artificial intelligence (AI) and machine learning models for clinical applications from large research medical centers or based on limited de-identified datasets. The introduction of transfer learning has resulted in AI models being accessible to researchers with fewer computational resources and less data, allowing them to leverage existing knowledge. Transfer learning is the ability to apply a model trained on a dataset to another dataset of interest.  A major consideration for transfer learning is domain shift, the dissimilarity between data distributions from the source used for training and the population to which the models are applied. This divergence between publicly available datasets and private, institution-specific datasets can introduce substantial bias and hinder the generalization of machine learning models to real-world scenarios.

In this study, we quantify the domain shift between the public domain (MIMIC-CXR) and a private dataset (VA-CXR). The efficacy of the transfer learning approach for chest X-ray classification and the subsequent impact on classification accuracy when dealing with domain shift form the main focus of this study.  The domain shift is traditionally viewed only based on the model and its performance, ignoring the multi-fold causes leading to the shift. We comprehensively address the effects of domain shift in this three-stage study: 1) Compare the performance between the source domain and target domain accuracy on the chest X-ray classification. 2) Quantify the quality of the ground truth extracted from the radiology reports, as supervised learning models are heavily dependent on the quality of the labels.  3) Analyze the relationship between demographic factors and classification accuracy, as domain mismatches often originate in demographic mismatches. 

By addressing the critical interplay of domain shift and demographic factors, this study not only provides insights into the technical challenges of adapting machine learning models to private datasets but also underscores the broader implications of these challenges in the context of healthcare. Our research contributes to developing more accurate, robust, and generalizable chest X-ray classification models by creating a systematic approach for using an existing model and understanding the nuances of a model's performance before applying it to a new population. 




§ METHODS





 §.§ Dataset

MIMIC-CXR is a publicly available dataset of 377,110 chest X-rays associated with 227,827 imaging studies from 65,379 patients. MIMIC-CXR is collected from the inpatient setting of Beth Israel Deaconess, a Boston hospital. (refer Table <ref>) We also use the test-split of this dataset (Test-split MIMIC-CXR) as the hold-out set analysis, which consists of 5159 studies with 293 patients <cit.>. 

VA-CXR is a private dataset of 259,361 chest X-rays associated with 91,020 imaging studies from 35,771 patients. VA-CXR is collected in the outpatient setting of the Boston Veterans Healthcare Administration station (refer to Table <ref>). The ground truth labels were extracted from the VA's corporate data warehouse (CDW) and joined with images using patient information from DICOM headers as published in Knight et al. <cit.>.   






 §.§ Ground truth Label Extraction


We used the CheXbert labelers <cit.> to expertly assign labels to 14 specific labels (Atelectasis, Cardiomegaly, Consolidation, Edema, Enlarged Cardiomediastinum, Fracture, Lung Lesion, Lung Opacity, No Finding, Pleural Effusion, Pleural Other, Pneumonia, Pneumothorax, Support Devices) associated with different chest conditions from radiology reports. CheXbert is a BERT-based approach that automates the detection of these observations, effectively streamlining the process of annotating medical images and reports. The NLP label extraction outputs scores for four classes: positive, negative, blank, and uncertain, associated with each of the 14 labels. As the class names indicate, for example, for Pneumonia, positive class: radiology report indicates that the patient has Pneumonia; negative class: radiology report indicates that the patient doesn't have  Pneumonia; uncertain class: radiology report mentions  Pneumonia but NLP tool unable to determine if it is positive or negative; blank class: radiology report doesn't mention  Pneumonia.



 §.§ Label Validation

Because ground truth assignment ultimately determines the accuracy of the imaging classifiers, we developed an image evaluation procedure in two steps. First, we evaluated the agreement between the NLP label extraction tool, CheXbert, and its precursor, CheXpert <cit.>, a rule-based tool. We focused on positive class agreements for our evaluation, using only positive classes for classification, and have combined uncertain/negative class agreements. The disagreement between NLP label extraction tools indicates ambiguity/ less confidence on the labels, ultimately creating an unreliable ground truth. The agreement is measured for both MIMIC-CXR and VA-CXR datasets; Of note that neither of the datasets was used to train the NLP extraction tools. 




  §.§.§ Relation to Diagnoses Codes

To validate the ground extracted from chexpert-labeler, we analyzed the relationship of specific ground truth labels to ICD codes in the patient's electronic health record (EHR) extracted from the VA's Corporate Data Warehouse (CDW).The assignment of ICD-9 and ICD-10 diagnosis codes associated with each condition was exploratory and not extensively optimized. 

Starting concepts were retrieved from the Chexpert-labeler github repository , where phrases they used to search notes can be found at https://github.com/stanfordmlgroup.  These phrases, along with our own expertise, were used to identify diagnosis codes and cross-reference radiology reports with diagnoses.For example:   Pneumonia was identified by chexpert-labeler as indicated by pneumonia, infection, infected process, and infectious; 
Edema was indicated by terms edema, heart failure, chf, vascular congestion, pulmonary congestion, indistinctness, and vascular prominence; Fracture was indicated solely by the word fracture; and Pneumothorax was identified by either pneumothorax or pneumothoraces.

This method enabled us to validate the ground truth labels by correlating them with the relevant ICD codes in the patients' EHRs, ensuring accurate cross-referencing of radiology reports with diagnoses.

For instance, the ICD-9 codes we used to indicate a pneumonia diagnosis in the outpatient diagnosis tables ranged from 480 to 486 and included 487.0. These codes encompass viral, bacterial, and other types of pneumonia, as well as pneumonia caused by unspecified pathogens. A similar approach was applied to ICD-10 codes for pneumonia and other conditions. The specific ICD codes used for each condition are detailed in Appendix Table <ref>.



 §.§ Multi-label Image Classification


Using the 14 labels extracted from the corresponding radiology reports with CheXbert, we created a multi-label image classification model from X-ray images. We used a pre-trained DenseNet model<cit.> as the core framework, removed the top classification layer, and integrated a custom classification layer for multi-label output. 
Previous work shows the effectiveness of different resolutions of DenseNet-121 based multi-label classification chest X-ray model on MIMIC-CXR dataset  <cit.>. This work uses the MIMIC-CXR trained DenseNet-121 Model on Chest X-ray pre-processed into 256x256 JPG images <cit.>. 




 §.§ Metrics

We evaluated our models using the Area Under the Curve (AUC). The AUC score was calculated separately for each of the 14 labels, indicating the separability measure for a given chest X-ray label. 
We also analyzed the difference in AUC scores between MIMIC-CXR and VA-CXR, and the prevalence for each label was calculated as the number of studies with positive results for a given label divided by the total number of labels, indicating the label's presence in the given cohort. 




 §.§ Domain Shift Analysis

We compared our source and target datasets along different dimensions: 1) Demographic Details: Age At Time of Imaging Study, Sex; 2) Imaging Study Details: Study Year, View  Point (Lateral view (Lat), erect anteroposterior (AP), posteroanterior (PA)) ; 3) Ground truth Labels: 14 labels. All the factors were analyzed against the accuracy of multi-label image classification. 
The impact can be estimated by comparing the prevalence of the above-listed factors across source and target domains with the performance of the multi-label classification. 

The subgroup analysis is performed on unseen datasets: Test Split MIMIC-CXR and VA-CXR. 


Study Year Analysis: We obtain the study year from VA-CXR studies date reported in DICOM headers.  The MIMIC-CXR dataset is unidentified; study years are replaced by anchor years. The comparison of performance across study years between the datasets isn't possible, so we don't present a study year analysis on the MIMIC-CXR dataset. 


Performance Analysis based on Sex: We obtain the sex of the patients in VA-CXR from the VA's corporate data warehouse (CDW) and the patients in MIMIC-CXR from the metadata. Sex is only classified into binary classes of male and female. We analyze the label-wise performance for each sex across both datasets. 


View Position Analysis: We obtain the view position of the images from the DICOM metadata. We use four main viewpoints for analysis: Lateral view (Lat), erect anteroposterior (AP), posteroanterior (PA))  and Left Lateral view (LL). We compare the label-wise performance across both datasets


Age Group Analysis:
We define the age of the patient as the age when the imaging study was performed. For VA-CXR, this age was calculated from the date of birth obtained from VA CDW and the date of imaging study from DICOM metadata. For MIMIC-CXR, we calculated age using the anchored date of birth and anchored imaging study date. The anchored dates were amended in a manner that the difference in years is constant, so we were able to approximate the age of the patient. The label-wise performance for both datasets was compared across six age groups: (0-50], (50-60], (60,70], (70,80], (80,90], and (90,100]. 



§ RESULTS




 §.§ Ground truth Label Validation

Table <ref> presents a comprehensive analysis of agreement and disagreement rates between Chexpert and Chexbert on the MIMIC-CXR and VA-CXR datasets across the 14 labels. The results shed light on the performance and consistency of Chexpert and Chexbert and the uncertainty of the ground truth labels. Notably, in MIMIC-CXR, Atelectasis was identified in 19.5% of cases with a disagreement rate of 10.1%, whereas in VA-CXR, the identification rate was lower at 9.8% with a disagreement rate of only 0.7%. This discrepancy in positive identification and disagreement rates is further exemplified in conditions like Cardiomegaly, where MIMIC-CXR reported positive identification in 15.7% of cases with a significant disagreement rate of 39.1%, contrasting with VA-CXR's 9.3%  positive identification rate and 13.7% disagreement rate.









 §.§ Comparison with Diagnosis Codes


While the X-ray classification accuracy was evaluated against the label extracted using NLP of the appropriate radiology report, it is possible also to compare directly against diagnosis codes in the clinical record.  However, interpreting this comparison is challenging due to two main issues: first, diagnoses are based on a broader range of information beyond just the X-ray, and second, patients may have multiple conditions, making it difficult to determine the specific reason each X-ray was ordered. Despite these challenges, we have conducted two useful comparisons of X-ray findings against diagnosis codes for nine categories within our dataset.

The top of Figure <ref> shows the fraction where a diagnosis related to the label was recorded within a week (either before or after) of the X-ray, compared to the total number of patients, including those who never had the diagnosis. Patients who had the diagnosis but not within the one-week window were excluded from this calculation. This plot, labeled 'Sensitivity,' reflects the proportion of cases where a diagnosis occurs within a week of noting the condition in the X-ray.  
We observe a positive finding on an X-ray, as extracted from the radiology report, which is mostly associated with a specific diagnosis of pneumonia and edema and frequently associated with pleural effusion and pneumothorax.  We also see that the ChexBert model is more frequently associated with a corresponding diagnosis than the ChexPert model.



The bottom of Figure <ref> compares the factor by which the enrichment ratio of positive finding in the radiology report increases with a concurrent (within one week) diagnosis.  Pleural effusion, for example, is seen 10 times more frequently in the radiology report when a concurrent diagnosis code is noted.  It is also true, however, that 2/3 of the pleural effusion diagnoses are not accompanied by a positive finding in a radiology report, as shown in the top bar chart, and also that 90% of the time a negative finding is made, no diagnosis is found.  We see that the conditions with the highest enrichment ratios differ from those with the highest sensitivities, but the trend that the ChexBert model generally outperforms the ChexPert model by a few to 20% remains.

Several factors were observed to contribute to the quantitative comparison of labels extracted from the radiology reports to the observed diagnoses, including the extent to which diagnosis codes could be found that correspond well to the radiology report finding, whether the X-ray is a screening or confirmatory test, the prevalence of the condition, and the chronic vs. acute nature of the condition.  We provide this figure as a survey across these factors and present the complete set of counts in Appendix Table <ref>.  Specific ICD 9 and 10 codes used for each condition are provided in the Appendix Table <ref>.





 §.§ Multi-label Image Classification on VA-CXR across NLP tools


Table <ref> compares ground truth extraction techniques, specifically focusing on the label-wise performance metrics on the VA-CXR dataset. Two techniques, ChexPert and ChexBert, are evaluated based on their Area Under the Curve (AUC), prevalence, and count across 14 labels. Across the findings, both techniques generally demonstrate similar AUC values, indicating comparable performance in distinguishing positive cases. For instance, in identifying Atelectasis, both ChexPert and ChexBert exhibit AUC values around 0.80, with similar prevalence and count numbers. Notably, in the case of Cardiomegaly, ChexBert shows a notably higher AUC of 0.862 compared to ChexPert's 0.753, suggesting potentially superior performance in this specific finding. However, the prevalence and count metrics vary across the findings and between the two techniques. For example, ChexPert tends to have higher prevalence and count values for several findings like ECM (Enlarged Cardiomediastinum), while ChexBert shows higher values for others such as Pleural Effusion. 




 §.§ Multi-label Image Classification Performance across multiple datasets


Table <ref> shows the comparison of MIMIC-CXR (Source dataset), Test Split MIMIC-CXR (Hold-out Source Dataset), and VA-CXR based on AUC on the 14 labels.  The Hold-out Source dataset is the test split of MIMIC-CXR dataset,  DenseNet-121 Model. The Test Split of MIMIC-CXR gives us a fair comparison to VA-CXR, the unseen target dataset. This can be observed based on the AUC drop from the overall MIMIC-CXR to test-split. In Table <ref>, the difference in AUC between Hold-out and Target indicates the performance variation between VA-CXR and Test Split of MIMIC. The Negative value of the difference in AUC indicates that the VA-CXR performs better than the Test Split MIMIC-CXR, and the positive value indicates that the Test Split performs better. The Enlarged Cardiomediastinum (ECM) label has the highest difference in AUC, indicating a huge performance drop in VA-CXR. This could directly impact the lack of a large number of image studies in ECM in the source dataset compared to the target. 






 §.§ Study Year-wise Performance on VA-CXR 

Table <ref> shows the label-wise distribution of the study years. The AUC systematically drops as for study year 2020 to 2022 for all labels except Consolidation and Pleural Other, which peaks at 2020. The prevalence increases over the years in VA-CXR for Atelectasis, Enlarged Cardiomediastinum, and Pleural Effusion. 








 §.§ Performance across datasets based on Sex

Figure <ref> compares the label-wise AUC and prevalence of the two sexes across the Test Split MIMIC-CXR and VA-CXR. This comparison is essential as the female-male patient ratio in VA-CXR is higher than that of MIMIC-CXR; dashed lines in the figure <ref> can observe this. 







 §.§ View Position-wise across datasets

Figure <ref> compares labels across Test-Split MIMIC-CXR and VA-CXR. The VA-CXR doesn't contain any Lateral images; the figure shows only MIMIC-CXR performance on the Lateral. The prevalence and AUC of viewpoints vary based on the label, with ECM having the most difference between the datasets. Pleural Other has a drop in performance in VA-CXR in AP view position, potentially due to low prevalence in both datasets. 
 






 §.§ Age Group-wise comparison across datasets

Figure <ref> shows the VA-CXR prevalence increase as the age increases across all labels except No Finding, Lung Lesion, and Pneumothorax.  The highest performance drops of 0.15 to 0.2 AUC  in VA-CXR compared to the Test Split of MIMIC-CXR can be observed in Enlarged Cardiomediastinum and Support Devices. In VA-CXR, it is interesting that the AUC performance across the age groups is more stable, i.e., there is not much change in AUC, with the exception of Atelectasis.  








 §.§ Summary

As seen from the results, the ground truth validation and multi-label classification performance across the NLP extraction tools showed that though the VA-CXR dataset has lesser disagreement rates than the MIMIC-CXR datasets, and there were AUC differences between models when using ChexPert and ChexBert (potentially propagated by distribution differences). When comparing the multi-label classification performance on different datasets, the unseen datasets didn't show domain shift other than a few labels such as Enlarged Cardiomediastinum. Among the different subgroup analyses, the study year had the most drastic differences in the performance of the multi-label classification model. These differences indicate that the domain shift is definitely of concern in study years.




§ DISCUSSION


Our study quantified the domain shift between a publicly available critical care dataset (MIMIC-CXR) and an outpatient, private institutional dataset (VA-CXR), assessing the efficacy of transfer learning for chest X-ray classification.  Because domain shift encompasses multifaceted factors, our comprehensive study was structured into three interrelated parts: the quality of ground truth, performance comparison, and subgroup analysis. 

Quality of Ground Truth 
Supervised learning models heavily rely on the quality of labels for training. Therefore, we quantified the ground truth quality extracted from radiology reports in both datasets. Despite being a source dataset, MIMIC-CXR exhibited significantly higher disagreement rates between the two ground truth information extraction NLP algorithms than the target, VA-CXR dataset. The potential performance drift of the target can be attributed to the mismatch of ground truth extraction methods. Our analysis underscored the importance of high-quality annotations in mitigating the effects of domain shift <cit.> and improving model performance.

Performance Comparison: 
We compared the difference in classification performance between MIMIC-CXR and VA-CXR to understand the extent of domain shift and its implications for model generalization. Our findings revealed notable variability in classification accuracy, highlighting the challenges posed by domain shift. We observe that the prevalence and performance were directly associated; for example, MIMIC-CXR's enlarged cardio mediastinum low prevalence may be the reason for its low performance. 

Subgroup analysis 
Subgroup analysis using demographic factors is crucial as we expect our populations to have different demographic distributions <cit.>. 
We observed a decline in the performance of the VA-CXR dataset over time, particularly in the years following 2020. This drop may be due to differences in the study years between the source dataset (MIMIC-CXR) and the VA-CXR dataset, as well as potential impacts from the pandemic years (note: we did not evaluate additional labels related to the pandemic).
We observed that both groups, though with different distributions, have aging populations across conditions of interest. Still, the prevalence of most conditions with age increases in the VA-CXR dataset, but it is highly variable in MIMIC-CXR.  For sex subgroup analysis, the performance was similar across the female and male populations in the datasets. Though the VA-CXR population is skewed towards the male population, the model performed well on the female population. This can be attributed to the source domain (MIMIC-CXR) having a balanced male-to-female ratio. 
For the view-based subgroup, we observed high variability between the views. This behavior can be attributed to the necessity of specific views for accurately diagnosing certain diseases. For instance, conditions such as Pneumonia and ECM primarily rely on PA (Posteroanterior) or AP (Anteroposterior) views for diagnosis.

Chest radiography is a first-line imaging modality for assessing lung and heart conditions. With advanced artificial intelligence techniques, the automation of chest X-ray abnormality detection has seen significant progress <cit.>. This has accelerated the pace of diagnosis and treatment and opened avenues for exploring biomedical applications in various clinical settings. However, the efficacy of these machine learning models heavily relies on the quality and availability of training data <cit.>.


Clinical relevance
Despite the surge in artificial intelligence and machine learning applications in clinical settings, their adoption has been uneven, primarily due to disparities in funding<cit.>, expertise, and availability of computing resources. While academic medical centers have been at the forefront of AI adoption, the accessibility of such resources remains a challenge for smaller research institutions. Introducing transfer learning has mitigated some of these challenges by enabling researchers to leverage pre-existing models trained on publicly available datasets for their specific applications. However, transfer learning introduces the concept of domain shift, which poses a significant obstacle to achieving robust and generalizable models.

Potential Workflows
While we primarily focused on matching image classification to known findings typical of chest X-rays and extracted those from radiology reports using NLP, this may not be the best workflow with which image classification can be incorporated into a medical outcomes workflow.  Tiu, et al. have noted the difficulties and inaccuracies associated with training chest X-ray classifiers against such labels and demonstrated improved accuracy and efficiencies with a self-supervised workflow to train their image classification model.<cit.>.  Even more ambitiously, they argue in Reference <cit.> that all modalities of data should be co-optimized in a much broader framework, known as a foundation model.  Our observations in Figure <ref> that most of the X-ray findings do not map directly onto diagnoses support the idea that more flexible workflows may be required to optimally incorporate AI-based image classification into medical outcome prediction.

By systematically addressing the challenges of domain shift and demographic factors, our study contributes valuable insights to developing more accurate and robust chest X-ray classification models. Furthermore, our findings have broader implications for healthcare, emphasizing the importance of understanding and mitigating biases in machine learning applications to ensure equitable healthcare delivery. Our research serves as a foundation for future studies aimed at refining transfer learning techniques and enhancing the generalizability of machine learning models in clinical practice.

Limitations
It is crucial to acknowledge that the data for the two studies were collected under significantly different clinical settings, which may influence the outcomes. The MIMIC database primarily includes inpatient data, where patients may be more critically ill, as indicated by the prevalence of portable anteroposterior (AP) images that tend to be less precise. In contrast, the VA outpatient population likely represents individuals who are relatively less ill, with higher-quality image data collection due to the setting's focus on outpatient care. Although we were unable to assess these differences quantitatively, we hypothesize that the clinical setting could substantially impact the performance of our models, influenced by the severity of diseases and other contextual factors.





§ CONCLUSION

In conclusion, our study sheds light on the critical interplay between domain shift, demographic factors, and the efficacy of transfer learning in chest X-ray classification. By quantifying the domain shift between a publicly available dataset and a private institutional dataset, we have identified disparities in classification accuracy and highlighted the challenges posed by transferring models across domains. Moreover, our analysis of demographic factors underscores the importance of considering population diversity in model development to ensure equitable healthcare outcomes. Moving forward, addressing these challenges will require concerted efforts to improve data quality, develop robust transfer learning techniques, and enhance the generalizability of machine learning models in clinical practice. Ultimately, by fostering a deeper understanding of these complexities, our research paves the way for the development of more accurate, reliable, and accessible chest X-ray classification models with the potential to transform patient care and advance the field of medical imaging.



§ ACKNOWLEDGMENTS

This work is sponsored by the US Department of Veterans Affairs using resources from the Knowledge Discovery Infrastructure which is located at the Oak Ridge National Laboratory and supported by the Office of Science of the U.S. Department of Energy. This manuscript has been authored by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The US government retains and the publisher, by accepting the article for publication, acknowledges that the US government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript or allow others to do so for US government purposes. DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (
http://energy.gov/downloads/doe-public-access-plan
). 






 
 



§ APPENDIX:




 §.§ Age When Image was Taken versus Study Year














 Paper Title: Evaluating SAM2's Role in Camouflaged Object Detection: From SAM to SAM2.








Evaluating SAM2’s Role in Camouflaged Object Detection: From SAM to SAM2
    Lv Tang    Bo Li 

vivo Mobile Communication Co., Ltd, Shanghai, China 

luckybird1994@gmail.com, 
libra@vivo.com
    August 5, 2024
=====================================================================================================================



§ ABSTRACT 
The Segment Anything Model (SAM), introduced by Meta AI Research as a generic object segmentation model, quickly garnered widespread attention and significantly influenced the academic community. To extend its application to video, Meta further develops Segment Anything Model 2 (SAM2), a unified model capable of both video and image segmentation. SAM2 shows notable improvements over its predecessor in terms of applicable domains, promptable segmentation accuracy, and running speed. However, this report reveals a decline in SAM2’s ability to perceive different objects in images without prompts in its auto mode, compared to SAM. Specifically, we employ the challenging task of camouflaged object detection to assess this performance decrease, hoping to inspire further exploration of the SAM model family by researchers. The results of this paper are provided in <https://github.com/luckybird1994/SAMCOD>.




§ INTRODUCTION

In recent years, large language models (LLMs) <cit.> have sparked a revolution in natural language processing (NLP). These foundational models exhibit remarkable transfer capabilities, extending far beyond their initial training objectives. LLMs showcase robust generalization abilities and excel in a multitude of open-world language tasks, including language comprehension, generation, interaction, and reasoning. Inspired by the success of LLMs, vision foundational models such as CLIP <cit.>, DINOv2 <cit.>, BLIP <cit.>, and SAM <cit.> have also emerged. The introduction of these foundational models continues to drive researchers’ exploration in the field of computer vision.

Among these foundational models, SAM stands out as an influential model in the domain of image segmentation. Upon its introduction, it quickly receives widespread attention and is utilized across various fields. To further extend SAM’s applicability, SAM2 <cit.> is introduced, designed to handle both image and video segmentation tasks within a unified architecture. Compared to SAM, SAM2 has significantly improved in terms of promptable segmentation accuracy and running speed, further enhancing the capabilities of the SAM model family.


In this technical report, we use the task of camouflaged object detection (COD) as a case study to analyze the progress and compromises involved in transitioning from SAM to SAM2. We observe the following two points: 1) When the SAM2 model is provided with prompts for segmentation, its performance shows substantial improvement compared to SAM. 2) However, when operating in auto mode, where SAM2 and SAM are both tasked with perceiving all objects in an image without prompts, the performance of SAM2 significantly deteriorates. We hope that our findings will further encourage researchers to explore both SAM and SAM2.



§ EXPERIMENTS



 §.§ Datasets and Metrics

To validate the performance of SAM and SAM2, we evaluate its performance on three benchmark datasets, containing CAMO <cit.>, COD10K <cit.>, NC4K <cit.> and MoCA-Mask <cit.>. We use six widely used metrics: structure-measure (S_α) <cit.>, mean E-measure (E_ϕ) <cit.>, F-measure (F_β), weighted F-measure (F^w_β), max F-measure (F^max_β) <cit.>, and mean absolute error (MAE).



 §.§ Progress



Video Task. One of the biggest advancements of SAM2 over SAM is its application in video tasks. Therefore, we first test the potential of SAM2 in the VCOD task, selecting the MoCA-Mask as the evaluation dataset. Specifically, for the first frame of each video sequence, we randomly select three prompt points based on its corresponding ground truth to identify the target object for segmentation. The video sequence is then input into the SAM2, and the segmentation results are obtained as shown in Table. <ref>. SAM2 significantly surpasses state-of-the-art VCOD methods SLTNet <cit.> and TSP-SAM <cit.>.






Promptable Segmentation. We further evaluate the performance of the SAM and SAM2 on promptable segmentation. Inspired by the recent study <cit.>, we use MLLMs, Shikra <cit.> and LLaVA <cit.> to generate coordinates for camouflaged objects in images. These coordinates are then input into both SAM and SAM2 to produce the corresponding mask results. The performance is shown in Table. <ref>. it is evident that SAM2’s performance significantly exceeds that of SAM. Moreover, through this approach, the performance of zero-shot CoD methods has the potential to match or surpass fully supervised methods. These findings highlight the advancements made by SAM2.





 §.§ Compromise










Auto Mode. In auto mode, we evaluate the performance of SAM2 and SAM. The evaluation method is similar to that described in the technical report <cit.>, and the comparison includes 22 CoD methods: SINet <cit.>, C2FNet <cit.>, LSR <cit.>, PFNet <cit.>, MGL <cit.>, JCOD <cit.>, TANet <cit.>, BGNet <cit.>, FDCOD <cit.>, SegMaR <cit.>, ZoomNet <cit.>, BSANet <cit.>, SINetV2 <cit.>, FAPNet <cit.>, the extension version of C2FNet <cit.>, DGNet <cit.>, CubeNet <cit.>, ERRNet <cit.>, VST <cit.>, UGTR <cit.>, ICON <cit.> and TPRNet <cit.>. The performance is shown in Table. <ref>. From the table, it is apparent that in auto mode, SAM2 seems unable to segment potential camouflaged objects. To further illustrate this point, we have compiled statistics on the number of masks predicted by SAM and SAM2 in each dataset, as shown in Table. <ref>. We can see that the number of masks predicted by SAM is six to ten times that predicted by SAM2. As shown in Fig. <ref>, for the prediction of masks corresponding to a specific image, SAM2 not only lags significantly behind SAM in terms of quantity, but also in quality.










§ CONCLUSION

This technical report has examined the transition from SAM to SAM2, showcasing both significant advancements and notable limitations. SAM2, an evolution from the foundational SAM, has proven to excel in tasks involving prompt-driven segmentation, where it outperforms SAM with enhanced accuracy and speed, particularly in handling both video and image segmentation. These improvements underscore SAM2’s potential as a versatile tool in the evolving landscape of vision-based models. However, our evaluations, particularly through the lens of camouflaged object detection, highlight a critical area where SAM2 lags behind its predecessor: operating in auto mode without prompts. Here, SAM2’s performance significantly declines, indicating a reliance on prompts that may limit its utility in scenarios demanding autonomous object recognition.
Therefore, we hope continued exploration and refinement of the SAM  family, aiming for advancements that retain the strengths of SAM while overcoming the limitations observed in SAM2. By addressing these challenges, we can push the boundaries of what foundational models can achieve in the realm of computer vision.


 





 

















ieee_fullname





 Paper Title: High-throughput 3D shape completion of potato tubers on a harvester.


1
.001


High-throughput 3D shape completion    


Blok et al.  


mode = title]High-throughput 3D shape completion of potato tubers on a harvester  



[1] 



[1]This study is partially supported by the Sarabetsu Village "Endowed Chair for Field Phenomics" project in Hokkaido, Japan. 


1]Pieter M. Blok[type=editor,
      orcid=0000-0001-9535-5354]
[1]
Conceptualization, Methodology, Software, Data curation, Writing - original draft


2]Federico Magistri[type=editor,
      orcid=0000-0003-2815-5760]
Methodology, Software, Writing - review & editing


2]Cyrill Stachniss[type=editor,
      orcid=0000-0003-1173-6972]
Methodology, Software, Writing - review & editing


1]Haozhou Wang[type=editor,
        orcid=0000-0001-6135-402X]
Data curation, Resources, Software, Writing - review & editing


1]James Burridge[type=editor,
        orcid=0000-0002-2194-9894]
Conceptualization, Methodology, Data curation, Supervision, Writing - review & editing


1]Wei Guo[type=editor,
        orcid=0000-0002-3017-5464]
Conceptualization, Methodology, Funding acquisition, Project administration, Supervision, Writing - review & editing


[1]organization=Graduate School of Agricultural and Life Sciences, The University of Tokyo,
            addressline=1-1-1 Midori-cho, 
            city=Nishitokyo-city,
            postcode=188-0002, 
            state=Tokyo,
            country=Japan
            
[2]organization=Center for Robotics, University of Bonn,
            addressline=Nussallee 15, 
            city=Bonn,
            postcode=53115, 
            state=North Rhine-Westphalia,
            country=Germany


[1]Corresponding author: pieter.blok@fieldphenomics.com (P.M. Blok).





§ ABSTRACT 
Potato yield is an important metric for farmers to further optimize their cultivation practices. Potato yield can be estimated on a harvester using an RGB-D camera that can estimate the three-dimensional (3D) volume of individual potato tubers. A challenge, however, is that the 3D shape derived from RGB-D images is only partially completed, underestimating the actual volume. To address this issue, we developed a 3D shape completion network, called CoRe++, which can complete the 3D shape from RGB-D images. CoRe++ is a deep learning network that consists of a convolutional encoder and a decoder. The encoder compresses RGB-D images into latent vectors that are used by the decoder to complete the 3D shape using the deep signed distance field network (DeepSDF). To evaluate our CoRe++ network, we collected partial and complete 3D point clouds of 339 potato tubers on an operational harvester in Japan. On the 1425 RGB-D images in the test set (representing 51 unique potato tubers), our network achieved a completion accuracy of 2.8 mm on average. For volumetric estimation, the root mean squared error (RMSE) was 22.6 ml, and this was better than the RMSE of the linear regression (31.1 ml) and the base model (36.9 ml). We found that the RMSE can be further reduced to 18.2 ml when performing the 3D shape completion in the center of the RGB-D image. With an average 3D shape completion time of 10 milliseconds per tuber, we can conclude that CoRe++ is both fast and accurate enough to be implemented on an operational harvester for high-throughput potato yield estimation. Our code, network weights and dataset are publicly available at <https://github.com/UTokyo-FieldPhenomics-Lab/corepp.git>. *














 3d shape completion potato deep learning rgb-d structure-from-motion


[
    [
    August 5, 2024
==================






§ INTRODUCTION


Potatoes (Solanum tuberosum) are important for global food security and can complement a cereal-based diet due to their high nutritional value and high yield  <cit.>. To further optimize potato yields, farmers must have sub-field level information on field yields at a 5-10 meter scale, as this provides important insights into the marketable yield of the field, as well as to further optimize the fertilization and cultivation practices. Sub-field level potato yield monitoring can be performed by systems installed on the harvester that can automatically estimate key yield parameters such as tuber number, size, volume, and weight. A commonly used potato yield monitoring system uses load cells attached to the harvesters' conveyor belt to measure the mass of the harvested produce in real time <cit.>. Such weighing systems are simple to use and maintain, but a disadvantage of these systems is that tare (such as stones, soil and crop residue) is also included in the weight measurement, making the tuber yield profile unobtainable.

A more promising alternative is to use a camera system that can visually distinguish potato tubers from tare, allowing the system to count and measure only the potato tubers. Currently, at least 14 of such camera systems have been presented in scientific literature: <cit.>. Most of these systems (11 of the 14) used a single red, green and blue (RGB) color camera to detect the potato tubers and then estimate their yield parameters using offline calibrated pixel-to-world conversion factors. While this method proved effective in calibrated laboratory setups, it was also found to have limited accuracy for potatoes that are occluded. Occlusion is a common phenomena on the conveyor belt of an operational harvester.

More recent studies have focused on extending the traditional RGB camera with additional three-dimensional (3D) vision. With 3D vision, it is possible to create partial or complete 3D shapes of potato tubers, allowing for better estimation of tuber yield under the challenging conditions of occlusion. <cit.> extended an RGB camera with a laser line triangulation method to capture the complete 3D shape of potato tubers. Although their system enabled very accurate volumetric estimates of potato tubers in a laboratory setup, the image acquisition method was unfortunately too time-consuming to be applied on an operational harvester. RGB cameras with embedded depth sensing abilities, known as RGB-D cameras, allow much higher throughput than laser triangulation methods. RGB-D cameras, such as the ones used by <cit.> and <cit.>, are therefore more promising for use on an operational harvester. 

A current limitation with RGB-D cameras is that they can only produce a partial 3D shape of the potato tuber, which can lead to an underestimation of the actual size, volume or weight. Combining multiple RGB-D cameras can potentially reduce the effect of this problem, but this will not fully solve it and make the overall system complex and more expensive. Therefore, it is more desirable and economically feasible to use a single RGB-D camera, and then estimate the complete 3D shape with dedicated software. There are numerous examples in the scientific literature of using 3D shape completion software to estimate complete shapes from partial shapes. Most of the current shape completion methods use deep learning techniques based on multi-layered perceptrons, graph-based convolutional neural networks and encoder-decoder networks to complete the 3D shape from partially completed point clouds <cit.>. Also in the agricultural domain, there are a few studies on 3D shape completion: one on the completion of plant leaves <cit.>, one on the completion of trees <cit.>, and five on the completion of fruits <cit.>. 

When assessing the applicability of 3D shape completion methods for potato yield estimation, it is crucial to consider their processing speed. Ideally, the shape completion method should be fast enough to process all the potato tubers that move over the conveyor belt during harvest, because this will provide the most complete and accurate yield estimate possible. Since roughly more than 200,000 potato tubers are harvested per hectare, the 3D shape completion method must complete processing in a matter of milliseconds. Of the studies mentioned above, only the method of <cit.> has the potential to do so. Magistri's method, called CoRe (Completion and Reconstruction), can complete 3D shapes in 4 milliseconds (tested on sweet peppers and strawberries). The novelty of CoRe is the addition of a convolutional encoder to the deep signed distance function decoder (DeepSDF, ). CoRe's convolutional encoder compresses RGB-D images into a latent vector, which is a compact representation of the shape's geometry. The latent vector from the encoder is then used along with the 3D coordinates of the point cloud as input for DeepSDF to reconstruct the complete 3D shape. Because the latent vector is predicted by the convolutional encoder, there is no need for DeepSDF's original time-consuming latent vector optimization, making the entire 3D shape completion method significantly faster.

Although the work by <cit.> shows the potential for high-throughput 3D shape completion, there are still unanswered research questions. First, what is the optimal size of the latent vector for implicitly learning the shape of the potato tubers? Second, given that the potato tubers are transported on the conveyor belt and thus move from bottom to top in the image, what is the best image location to perform the 3D shape completion? Third, and most important, is the 3D shape completion method able to quickly and accurately estimate the volume of fast moving potato tubers on an operational harvester?

The main contribution of this paper is the extension of the work by <cit.> to answer these research questions. We have conducted a systematic study and analysis on how to optimize this 3D shape completion method. Our optimization has led to several improvements to CoRe's original convolution encoder: an improved data preprocessing, geometric and color data augmentations, an updated loss function, an updated neural network architecture, and a graphical processing unit (GPU)-based 3D mesh generation that improves and speeds up the 3D shape completion. 

This paper presents the research and development of a 3D shape completion network for estimating the volume of individual potato tubers on an operational harvester. Our research novelties are four-fold. First, we optimized CoRe's original convolutional encoder for faster and more accurate 3D shape completion of potato tubers on an operational harvester. Second, we performed a systematic analysis of the effect of the latent size, image analysis region, potato size, and potato cultivar on the performance of the 3D shape completion of potato tubers. Third, we conducted two ablation studies on the impact of our new additions to the overall performance. Fourth, we publicly released our 3D dataset with partially and fully completed point clouds of potato tubers collected on an operational harvester in dirty and cluttered circumstances. 



§ MATERIALS AND METHODS

 



 §.§ Dataset





  §.§.§ Imaging system

We installed an imaging system above the conveyor belt of a single-row potato harvester (Toyonoki Top-1, Figure <ref>). The imaging system was a black plastic box in which four light emitting diode (LED) strips were installed to provide light (Figure <ref>). Reflective curtains on the sides of the plastic box helped to diffuse the light (Figure <ref>). In the center of the box, an RGB-D camera (Intel Realsense D405) was installed. The distance between the RGB-D camera and the conveyor belt was approximately 0.33 m. At this distance, the camera's field of view was 0.64 m (width) by 0.39 m (height). The RGB-D camera was connected to a laptop computer (Lenovo P51) that used the Robot Operating System 2 (ROS2, version: Humble Hawksbill) to collect color and depth images with 30 frames per second in ROS2 bag files. 





  §.§.§ Image collection on the harvester

RGB-D images were collected from 12 rows in a potato field in Sarabetsu, Japan (latitude: 42.610316, longitude: 143.156753). The row spacing in this field was 0.75 m and the potato cultivar was Sayaka. For each row, a separate ROS2 bag file was recorded and stored on the laptop. To increase diversity in potato sizes and shapes, we also collected data by running the harvester in the barn of the farm and then dumping boxes of two different potato cultivars on the conveyor belt. Although this mimicked the density of potato tubers on the conveyor belt in the field, there was no tare in the boxes, making the detection task easier. The barn experiment was conducted with six boxes of potatoes, each weighing between eight and ten kilograms. The six boxes were divided into three boxes with potatoes from the Kitahime cultivar and three boxes from the Corolle cultivar. A separate ROS2 bag file was recorded for each box and stored on the laptop.

While running the harvester in the field or in the barn, we collected images of 339 potato tubers of different sizes and shapes to test our 3D shape completion method. Our collection method consisted of the following procedure: one person standing in front of the imaging box selected a potato tuber of a random size and shape and then inserted a colored thumbtack into the potato. The thumbtack was inserted into the potato such that it was visible in the RGB image when the tuber passed the image acquisition region of the RGB-D camera. Given the speed of the conveyor belt, we were able to capture between 20 and 30 images of the marked potato tuber as it moved under the camera along with the conveyor belt (Figure <ref>). After the image acquisition, another person standing behind the imaging box collected the marked potato by hand. The marked potato was then placed in a bucket that was later brought to the barn for 3D reconstruction. 





  §.§.§ 3D reconstruction of the collected potato tubers


To obtain the complete 3D shape of the collected potato tubers, we have set up an imaging system in the barn (Figure <ref>a). Our imaging system consisted of three Canon X7 DSLR cameras, a turntable, four auto-detectable marker stands (Fig. <ref>b), and a photo studio with LED illumination. 

Prior to taking the images, each collected potato tuber was pierced with a narrow threaded bolt attached to a white wooden board (Figure <ref>b). This piercing allowed the tuber to be photographed from three camera perspectives while being held off the ground, allowing almost the entire longitudinal section of the tuber to be photographed at once. To capture the different sides of the tuber, the turntable was set to automatically rotate 15 degrees and then stop for two seconds to give the three cameras time to acquire the images. This was repeated until the tuber was photographed from all sides. The image acquisition was accomplished with a commercial shutter controller (Esper TriggerBox) that released an electronic trigger to the three cameras at the same time. The images were directly transferred to the connected laptop computer to store them for later 3D reconstruction (Figure <ref>c). The camera parameter settings, image transferring, and image renaming were accomplished with DigiCamControl software, of which its details are shown in Figure <ref>d.



After the image acquisition, the images were processed to extract the region of the potato tuber. This region extraction was accomplished by applying a threshold on the CIELAB color space to filter out the white- and black-colored background (Figure <ref>e) (the threshold was set to larger than 15 for the A channel and B channel). The resulting mask was refined by using the pre-trained CascadePSP <cit.> deep learning network (Figure <ref>g&h).

After generating the masks of all tubers, we implemented an automatic batch processing workflow based on the Metashape API for 3D reconstruction (Figure <ref>i). First, the images and corresponding masks of each potato tuber were grouped for each camera and its corresponding view angles. Then, the control point markers in the images were automatically detected, the scale bars were automatically imported, and world coordinates were automatically assigned to each potato tuber. Afterwards, the images of the different cameras were aligned, and the corresponding key points were generated. These aligned images served as input for the 3D reconstruction, which resulted the 3D mesh model, including its colored model textures. After 3D reconstruction, small distortions, such as 3D meshes belonging to the white wooden board were removed. Then, the filtered meshes were double-checked by another researcher in CloudCompare who removed small disconnected components if necessary. Figure <ref> illustrates a filtered 3D mesh with colored textures of a potato tuber. 





  §.§.§ Dataset splits


After the 3D reconstruction of the 339 collected potato tubers, we split the dataset into a train, validation and test set. The split was made in such a way that the three sets contained a representative portion of different sizes, shapes and cultivars (Figure <ref>). Our split ratio was 70% for the train set (237 potato tubers), 15% for the validation set (51 potato tubers) and 15% for the independent test set (51 potato tubers). 

Because each potato tuber was photographed 20 to 30 times, the total number of RGB-D images for training the encoder was 6794. The number of validation images was 1439 and the number of independent test images was 1425.





 §.§ 3D shape completion network




  §.§.§ Encoder and decoder architecture




Our 3D shape completion network was based on the CoRe network presented in <cit.>. Given the many updates made to CoRe during our research, our 3D shape completion network was renamed to CoRe++. Figure <ref> gives a schematic overview of CoRe++. CoRe++ consisted of a convolutional encoder and a decoder. The encoder was a small neural network consisting of seven convolutional layers, each followed by a Leaky-ReLU activation function and a Max-Pooling layer, see the details in Table <ref>. The position of the pooling layer was one of the changes of CoRe++ compared to CoRe (in CoRe, the pooling layer was before the activation). We hypothesized that this positional change of the pooling layer could help prevent the loss of important features. After the last convolution block, there was a flatten layer after which the flattened output tensor was fed into a fully connected layer. This layer outputted the latent vector, whose dimension was made configurable for the purpose of our study. Conceptually, the latent vector is a compressed representation of the 3D shape of the potato tuber.



The decoder of CoRe++ was the coded-shape deep signed distance function (DeepSDF, ). This decoder was composed of nine fully connected layers with a feature dimension output of 512. The first fully connected layer had an input dimension of latent size + 3, which was the result of concatenating the latent vector with the 3-dimensional vector of the object's coordinates in x, y, z. After the last fully connected layer there was a ReLU activation function followed by a hyperbolic tangent function. The latter outputted values between -1 and 1. Conceptually, this output is the signed distance to the object's surface, in which positive values represent 3D points outside the object's surface and negative values represent 3D points inside the object's surface. The value's magnitude corresponded to the distances to the object's 3D surface, where values close to zero approximated the object's 3D shape. The concept of signed distances is visualized in Figure <ref>.





  §.§.§ Data input, preprocessing, augmentation, and postprocessing


The input for the CoRe++ network was a four-channel RGB-D image that was masked to the potato region and clipped to a fixed-sized image of 304 x 304 pixels (Figure <ref>). The image dimension was chosen such that the largest tuber in our dataset would completely fit in the image. The input boxes and masks were obtained after manually annotating the potato tuber regions with the LabelMe annotation software. Note that this clipping and masking can also be achieved with an object detection or instance segmentation model.

Two new data preprocessing techniques were added to CoRe++. The first was a depth pixel filtering algorithm, which was extracted from <cit.>, that removed all depth pixels that were further away from the depth values of the majority of the pixels in the masked depth image. This majority of pixels was assumed to represent the potato region, and helped to remove depth outliers in the depth image. The second preprocessing technique was a normalization of the remaining depth pixels between the minimum (230 mm) and maximum distance value (350 mm) between the RGB-D camera and the potato tubers on the conveyor belt. This normalization helped to increase contrast in the depth image, which potentially leads to better model convergence. 

In addition to these new data preprocessing techniques, we also added geometric and color data augmentations to the encoder training procedure. The geometric augmentations involved a random image rotation to a maximum of 45 degrees, a random horizontal flipping of the image and a random vertical flipping of the image. The geometric data augmentations were applied to both the RGB image and the depth image. The color augmentations were only applied to the RGB image, and involved a random change of the brightness, saturation and hue of the image. The parameters (0.5 for brightness, 0.5 for saturation and -0.1 to 0.1 for hue) were chosen such that the augmented image looked different but visually realistic compared to the original image. 

The data augmentation for training the DeepSDF decoder involved a random scaling of the original 3D shape with scaling parameters between 0.5 and 2.0, meaning that the original shape was scaled between half of its original size and double of its size. The second data augmentation involved a rotation of the 3D shape around the Z axis by a rotation value between 0.0 and 30.0 degrees. The last data augmentation was a random shear of the 3D shape of maximally 0.5 in the X direction. For each original 3D shape, 10 augmented 3D shapes were included into the train set. 

As Figure <ref> indicates, the output of the CoRe++ network is a completed 3D point cloud. Since it is not possible to calculate volume directly from a 3D point cloud, we added a 3D data postprocessing procedure that converted the 3D point cloud into a watertight 3D mesh from which the volume can be estimated. Our 3D mesh generation procedure consisted of selecting the predicted signed distance values less than or equal to 0.0. These values represented the 3D points on the surface and inside the potato tuber, refer to the blue- and red-colored points in Figure <ref>. From the selected points, a 3D convex hull shape was extracted using Open3D's GPU-accelerated hull function. The convex hull was then smoothed with a triangle-based algorithm <cit.> that divided each triangle of the hull into four smaller triangles. What followed were noise suppression procedures and an iterative voxel-based downsampling of the generated hull in case it was not watertight. The latter procedure guaranteed that each produced 3D mesh was watertight so that the volume could be calculated. 



  §.§.§ Training procedure


The encoder and decoder were trained in the opposite order: first the DeepSDF decoder was trained on the complete 3D shapes in order to optimize the latent vectors. Then, the encoder was trained to fit the preprocessed RGB-D images to the target latent vectors. In Figure <ref>, these two training procedures are visualized by the blue colored and purple colored arrows, respectively. 

At the start of the DeepSDF training, a randomly initialized latent vector was assigned to each data point. These latent vectors were then optimized along with the weights of the decoder using standard backpropagation. In our research, the DeepSDF decoder was trained for 1001 epochs with a step-based learning rate scheduler that started at a learning rate of 5·10-4. For every 300 epochs, the learning rate was halved. The decoder was trained with the Adaptive Moment Estimation (Adam) optimizer. We used the L1 loss function during training, because this loss function serves the purpose of minimizing the sum of all distance differences between the target 3D shape and the predicted 3D shape. For every 10 epochs, a weight file and a latent vector code were automatically saved. After training, the weight file with the best 3D reconstruction on the shapes of the validation set was selected. This selected weight file and the corresponding latent vector code were used as targets for training the encoder. The metric for determining the best 3D reconstruction of the DeepSDF decoder was the Chamfer distance. The Chamfer distance (d_CD) represents the sum of the average closest distance from points in the ground truth 3D shape (𝒢) to the points in the reconstructed 3D shape (𝒮), and vice versa (Equation <ref>). The lower the Chamfer distance, the better the 3D reconstruction.


    d_CD(𝒢, 𝒮) = 1/|𝒢|∑_x ∈𝒢min_y ∈𝒮 x - y ^2_2 + 1/|𝒮|∑_y ∈𝒮min_x ∈𝒢 y - x ^2_2


where |𝒢| and |𝒮| are the numbers of points in 𝒢 and 𝒮, respectively
 

The weights of the encoder were randomly initialized before training started. During encoder training, the weights of the decoder were frozen. The encoder was trained for 100 epochs. The initial learning rate was 1·10-4, and this value was gradually decreased by an exponential learning rate scheduler of 97%. The encoder was also trained with the Adam optimizer. We used the mean squared error loss function (MSE) to minimize the squares of the differences between the target latent vector and the predicted latent vector. We chose this loss function because it penalizes larger differences between the target and predicted latent vector more heavily than CoRe's default L1 loss function. Besides the MSE loss function, we used the contrastive loss function by <cit.>. The rationale is that contrastive loss encourages the encoder to learn latent vectors that are well-separated in the latent space for the different potato tubers. Contrastive loss also enforces the encoder to learn latent vectors closer in the latent space for the images belonging to the same potato tuber. The contrastive loss function is summarized in Equation <ref>, where N is the total number of potato tubers, 𝐳_i and 𝐳_j represent the latent vector of instances i and j, respectively, y_i and y_j denote the potato tuber identifiers of instances i and j, respectively, δ_rep is the margin parameter controlling the minimum separation between representations of instances with different tuber identifiers (in our research δ_rep was set to 0.5), · denotes the Euclidean distance between two latent vectors, ·_+ is the positive part of the argument, ensuring that only positive differences contribute to the loss.


    L_c = ∑_i=1^N∑_j=1^N𝐳_i - 𝐳_j _+,    if  y_i = y_j 
    max{0, δ_rep - 𝐳_i - 𝐳_j },    if  y_i ≠ y_j


The combined loss function for training the encoder is summarized in Equation <ref>. The loss contribution values were set to 1.0 for w_mse, and 0.05 for w_c. These values were optimized in a pre-comparative experiment.


    ℒ = w_mse·ℒ_mse + w_c·ℒ_c


For every epoch the encoder was inferred on the validation set with the decoder activated. The encoder weights with the lowest root mean squared error (RMSE, Equation <ref>) on the volume were stored for final evaluation on the independent test set. We chose the RMSE metric, because it was the most common metric in the scientific literature for evaluating the volumetric estimate of potato tubers. The ground truth volume (V, Equation <ref>) was extracted from the reconstructed 3D mesh (Section <ref>). This volume estimate was more accurate than the volume estimate of the ground truth method with water displacement, as the latter had an accuracy of 10 ml at best and was more prone to human reading and writing errors. 


    RMSE = √(1/n∑_i=1^nV̂ - V^2)

where V̂ and V are the estimated volume and the ground truth volume, respectively. n is the number of potato tubers in the dataset.



 §.§ Evaluation


The Chamfer distance (Equation <ref>, Section <ref>) was used as the performance metric for evaluating the 3D point cloud completion. Besides the Chamfer distance, we also calculated the precision, recall and the F-score. The precision (p) was the percentage of reconstructed points within a certain distance (d) to the ground truth point cloud (Equation <ref>). As such, it represented the accuracy of the 3D reconstruction. The recall (r) was the percentage of ground truth points within a certain distance (d) to the reconstructed point cloud (Equation <ref>). The recall represented the completeness of the 3D reconstruction. For our evaluation, we used a distance (d) threshold  of 5.0 mm, as this value allowed to compare our results one-on-one with those of <cit.>. After calculating the precision and recall, the F-score was obtained (Equation <ref>). The F-score was the harmonic mean between the precision and the recall, and it represented the percentage of 3D points that were correctly reconstructed.


    precision(d) = 100/|𝒮|∑_y ∈𝒮min_x ∈𝒢 y - x  < d



    recall(d) = 100/|𝒢|∑_x ∈𝒢min_y ∈𝒮 x - y  < d



    f-score(d) = 2 · p(d) · r(d)/p(d) + r(d)


For evaluating the volumetric estimate we used the RMSE metric, as presented in Equation <ref>. To evaluate the processing speed, we calculated the average 3D shape completion time on the test images. This analysis was performed with a Lenovo Legion Pro 7 16IRX8H laptop with a NVIDIA GeForce RTX 4090 Laptop GPU.



 §.§ Experiments




  §.§.§ The effect of the latent size on the 3D shape completion result


For 3D shape completion, it is important that the encoder-compressed latent vector contains high-level distinguishable 3D features that generalize well on new and untrained shapes. Because the latent vector plays such an important role in an encoder-decoder network like ours, we set up an experiment that tested what was the optimal size for the latent vector for completing the 3D shape of potato tubers. We investigated six sizes: 8, 16, 32, 64, 128, and 256. Latent size 32 was used in the original study by <cit.>. Our hypothesis was that the smaller sizes, such as 8 and 16, had insufficient capacity to compress the 3D shape, leading to suboptimal generalization. Also, we hypothesized that the larger sizes, such as 128 and 256, were probably too large resulting in insufficient compression to generalizable 3D shape features. Hence, we hypothesized that the medium sizes, 32 and 64, were probably more optimal for the compression and thus generalization of 3D shapes.  

To better understand the optimal size of the latent vector, we conducted an additional experiment with our CoRe++ network. This experiment consisted of interpolating the values of the latent vector of the smallest potato tuber and the largest in the test set. By interpolating the values between these two latent vectors, we were able to better understand the diversity of the generated 3D shapes and sizes in the corresponding latent space of the six tested latent sizes. Ideally, the latent space is constructed so that after interpolation, the generated 3D shapes are realistic in shape and have a sequential build-up in size, as the interpolation was performed between the smallest and largest tuber.  

The experiment was tested with 3 networks: the DeepSDF decoder-only architecture without the encoder, the original CoRe network by <cit.>, and our CoRe++ network. The DeepSDF decoder-only architecture was tested to better understand where the largest effect of latent size was: in the encoder or in the decoder. Testing the original CoRe network by <cit.> provided insight into a potential trend commonly shared between CoRe and CoRe++. It also gave a more detailed overview on the potential improvement of CoRe++ over CoRe. 

The performance parameters were the ones listed in Section <ref>. Our evaluation was based on two requirements: accuracy and analysis speed. The accuracy requirement was met if the RMSE on the volume was lower than that of a standard linear regression model. Input to that linear regression model were the length, width and depth estimates of the partially completed point cloud (Figure <ref>). This point cloud was obtained by converting the filtered RGB-D image (Figure <ref>) into 3D points in Open3D software using the intrinsic camera parameters of the RGB-D camera. From the generated point cloud, the oriented 3D bounding box was obtained from which the length, width and depth dimensions were extracted as input parameters for training the linear regression model. The linear regression model was trained on the same train and validation set and tested on the same independent test set (Section <ref>). The trained linear regression model resulted a RMSE of 31.1 ml on the volumetric estimate of the test images. As such, if the 3D shape completion method had a RMSE lower than 31.1 ml, then the accuracy requirement was met. 



The analysis speed requirement was met if the 3D shape completion was finished in less than 16 milliseconds (ms) per potato tuber. This analysis time was calculated based on the highest number of potato tubers present in a single image in our dataset: 59. The average throughput time of the potato tubers on the harvester was 0.95 seconds, and this value was derived by dividing the average number of 28.5 frames per tuber (9658/339, Section <ref>) by the camera acquisition rate of 30 frames per second. With the maximum number of tubers in a single image and the average throughput time, we estimated that the 3D shape completion method should analyze up to 62 potato tubers per second (59/0.95). This equaled to 16 ms per tuber (1000/62).



  §.§.§ The effect of the potato size, potato cultivar, and image analysis region on the 3D shape completion result


The goal of this experiment was to gather insights into the performance of our CoRe++ shape completion method when applied in the field. The first practical analysis was on the effect of the tuber size on the 3D shape completion result. This analysis is important because it provides information about the degree of generalizability of our method in a particular field. Under typical field conditions, a wide variety of tuber sizes move over the harvester's conveyor belt, which means that our 3D completion method should be able to generate accurate volumetric estimates regardless of the tuber size. We decided to perform the analysis on four different size classes, which were extracted from our dataset distribution (Figure <ref>): small tubers with a volume between 0 and 100 ml, small to medium tubers with a volume between 100 and 150 ml, medium to large tubers with a volume between 150 and 200 ml, and large tubers with a volume between 200 and 500 ml. The ranges were chosen so that a relatively balanced number of tubers ended up in the four classes. 

The second practical analysis was on the effect of the potato cultivar on the 3D shape completion result. This analysis provided insights into the degree of generalizability of the 3D completion method at the farm level, as the majority of farmers grow multiple cultivars in a season. We wanted to test if our method can provide accurate volumetric estimates regardless of the potato cultivar. We conducted the experiment with three cultivars (Corolle, Kitahime and Sayaka) that differed in shape and size. Corolle was the cultivar with the most elongated and smallest tubers. Kitahime was the cultivar with the most spherical and medium-sized tubers. Sayaka was the cultivar with the largest tubers and the greatest diversity in shape. This may be due to the fact that Sayaka was the cultivar with the most samples in our dataset. As shape metrics, we calculated the elongation factor and the concavity factor. The elongation factor was obtained by dividing the longest dimension of the 3D bounding box of the ground truth mesh by the shortest dimension. Spherical shapes have an elongation factor close to 1 and elongated shapes have an elongation factor closer to 2. The concavity factor was obtained by calculating the Chamfer distance between the original ground truth 3D mesh and the convex hull of that mesh. Higher Chamfer distances mean that there are more valleys or concave regions on the surface of the potato tuber. 

The final practical analysis was on the effect of the image analysis region on the 3D shape completion result. This analysis provided insight into the RGB-D image region where ideally the 3D shape completion should be performed. This analysis was necessary because it is computationally intensive to analyze all RGB-D frames for each tuber. We intended to identify an area in the RGB-D image where the RMSEs for the volumetric estimate are lowest on average. That particular area will be most promising for performing the high-throughput 3D shape completion. In our analysis, the RMSEs were summarized for thirteen horizontal regions in the RGB-D image (Table <ref>).






  §.§.§ Ablation studies

Two ablation studies were performed as a final experiment. The first ablation study was conducted on the additions to CoRe++, which helped us to better understand the impact of the individual additions to the overall performance. The second ablation study was conducted on the components that were commonly shared between CoRe++ and CoRe. This ablation study helped us to better understand the impact of changes in the input data and network architecture of the convolutional encoder. 

The first ablation study consisted of examining the impact of seven additions made to CoRe++. The first two examined ablations consisted of individually deactivating the depth normalization and the depth filtering when training the convolutional encoder (both data preprocessing steps are described in Section <ref>). After that, we examined the impact of deactivating the data augmentation when training the convolutional encoder. The fourth ablation examined the effect of repositioning the Max-Pooling layer before the LeakyReLU activation, as was the case in the original CoRe implementation (refer to Section <ref>). The fifth examined ablation was changing the MSE loss function back to the L1 loss function, as was the case in CoRe (refer to Section <ref>). The sixth examined ablation consisted of replacing CoRe++'s method for automatically determining the best network weights (using GPU-based 3D mesh generation and the RMSE volume metric) with CoRe's original method (using marching cubes <cit.> mesh generation and Chamfer distance metric). The seventh examined ablation consisted of deactivating CoRe++'s 3D smoothing technique which was described in Section <ref>. 

The second ablation study consisted of examining the effects of ten ablations in the input data and network architecture that are commonly shared between CoRe++ and CoRe. The first ablation involved training CoRe++ with a single-channel depth image instead of the original four-channel RGB-D image. This ablation gave us a better understanding of the effect of adding color channels to the input data for completing the 3D shape. The second ablation involved training CoRe++ with an RGB-D image clipped to the bounding box instead of the original mask. This ablation gave us insight into how the final application on the potato harvester should look like: one based on an object detection model or one based on an instance segmentation model. Logically, this ablation also gave insight into the future annotation effort, which would be significantly higher when using an instance segmentation model. The third ablation involved reducing the encoder's network architecture from seven to five convolutional blocks (layers 1-10 in Table <ref> followed by a flatten layer and a fully connected layer). This ablation gave us insight into the effect of using an even lighter network in the situation of hardware constraints. The fourth ablation involved removing the pooling layers (the even layers in Table <ref>) to investigate their effect on the overall performance. The fifth ablation involved replacing CoRe++'s original LeakyReLU activation with the standard ReLU activation. The sixth ablation involved disabling the contrastive loss element in the loss function so that only the MSE loss was used during training. The seventh to the tenth ablation involved increasing or decreasing the learning rate in increments of two and five. This resulted in using a learning rate of 5·10-4 (learning rate · 5), 2·10-4 (learning rate · 2), 5·10-5 (learning rate · 0.5) and 2·10-5 (learning rate · 0.2). 



§ RESULTS





 §.§ The effect of the latent size on the 3D shape completion result

Table <ref> summarizes the results of the 3D shape completion for the three 3D completion methods and the six latent sizes. Interestingly, the effect of the latent size was marginal when testing the DeepSDF decoder-only network. For this network, the smallest latent size of 8 resulted in the lowest Chamfer distance and RMSE on the volume. One possible explanation for this result is that the DeepSDF decoder uses an iterative optimization process during inference to extract the best possible latent vector. This iterative process probably helped to optimize the latent vector for each potato shape in the test set, making the 3D shape completion slow (33 seconds on average), but also better optimized for each of the six tested latent sizes. 

For both CoRe and CoRe++, the best latent size was 32, followed by 64. This result is consistent with our hypothesis that the medium-sized latent vectors are better for compressing the data into meaningful 3D representations. Figure <ref> shows that latent sizes 32 and 64 have the most diverse 3D shapes and sizes after latent space interpolation. Compared to the other sizes, latent sizes 32 and 64 have both spherical and elongated 3D shapes, and the sizes show a proper sequential build-up. Latent size 8 and 256 are both worse in 3D shape completion performance (Table <ref>) and latent space interpolation (Figure <ref>). Latent size 8 could only produce five valid 3D shapes out of the seven interpolated latent vectors, while latent size 256 failed to produce a sequential size build-up. Latent sizes 16 and 256 produced mainly elongated 3D shapes, meaning that they had a limited ability to produce the more spherical shapes. 

In terms of 3D shape completion and volumetric estimate, CoRe++ outperformed CoRe for five of the six latent sizes: 16, 32, 64, 128, 256 (Table <ref>). The difference between the 3D shape completion result of CoRe++ and CoRe is visualized in Figure <ref>. As for CoRe++, the reconstructed 3D shapes are smooth and they approximate the real shape of the potato tuber. With CoRe, the 3D shapes are rougher and spikier, leading to less resemblance to the real potato shape, larger Chamfer distances and larger volumetric errors. 

Only for CoRe++ and latent sizes 32 and 64, the accuracy requirement was met as the corresponding RMSE values were less than 31.1 ml (this was the baseline value when using linear regression). The requirement for analysis speed was met for each latent size for both CoRe and CoRe++, as all of the analysis times were less than 16 ms (Table <ref>).









 §.§ The effect of the potato size, potato cultivar, and image analysis region on the 3D shape completion result


Table <ref> summarizes the effect of the potato size on the 3D shape completion result. There is a trend that the RMSE on the volumetric estimate is larger when the tuber is larger. A possible explanation is that the larger tubers are more concave (Table <ref>), meaning that they have more valleys and variations in their 3D curvature, making it harder to accurately reconstruct the 3D shape (especially when the concave parts are facing downward with respect to the camera's perspective). The fact that larger Chamfer distances are observed on the largest tubers supports this explanation. For the relative volumetric errors, there is an opposite trend, meaning that the largest relative volumetric errors are observed on the smaller tubers. 



Table <ref> summarizes the effect of the potato cultivar, and thus indirectly the shape of the potato tuber, on the 3D shape completion results. There is a trend that the Chamfer distance is lower when the tubers have a more spherical shape (i.e. elongation factors closer to 1.0). For the RMSE on the volumetric estimate, there is no clear trend regarding the tuber shape. What may explain this result is that the different cultivars had different tuber sizes: the cultivar Corolle had the relative smallest tubers and it was already shown in Table <ref> that the RMSE is lowest when the tubers are smaller. The largest RMSE was observed on Sayaka, which was the cultivar with the largest tubers on average, but it was also the cultivar with the largest number of samples, meaning that there was probably more diversity in tuber shape and size.
 



Figure <ref> visualizes the RMSE on the volumetric estimate for the thirteen different image regions. The smallest RMSE of 18.2 ml was observed in the central horizontal region of the image between 350 and 400 pixels. The largest RMSE values were observed in the lower and upper parts of the image, indicating that these regions are not recommended for performing the 3D shape completion. These results may indicate that some form of lens distortion may have occurred in the peripheral regions of the camera's field of view, leading to poorer 3D shape completion in these regions. There may also have been a higher degree of occlusion in these regions.





 §.§ Ablation studies




The ablation study on CoRe++'s additions (Table <ref>) shows that the largest contribution to the overall performance was made by CoRe++'s validation method, which was based on GPU mesh generation and RMSE validation metric. Changing this validation method to the original validation method of CoRe with marching cubes mesh generation and Chamfer distance validation metric, led to the largest increase in both Chamfer distance (+32.1%) and RMSE on volumetric estimate (+122.1%). An obvious explanation for this result is that the final volumetric estimate benefits from having the RMSE optimized during training. Another explanation is that CoRe's original marching cubes method seems unable to accurately reconstruct the 3D shapes of the potato tubers. This may be due to the chosen grid density of the marching cubes method which was optimized for high-throughput 3D reconstruction, but possibly came at the expense of the accuracy (something that can also be observed in Figure <ref>). Three other additions to CoRe++ that significantly improved the overall performance were the two data preprocessing steps (highlighted by the first and second ablation) and the loss function modification to MSE (highlighted by the fifth ablation).

Regarding the generic ablation study (Table <ref>), there are a few interesting outcomes. First, the choice of the activation function has a large impact on the overall performance. The fifth ablation shows that after replacing the LeakyReLU activation with a standard ReLU, the Chamfer distance increased by 46.4% and the RMSE by 96.0%. This outcome is consistent with that of <cit.>, who also found that ReLU under-performs compared to the more advanced variants of ReLU when testing an autoencoder on the Global Wheat Head dataset <cit.>. In our case, we think that the LeakyReLU activation resulted in a better and faster network convergence during training, due to LeakyReLU's ability to maintain non-zero gradients for negative inputs. A second interesting outcome was the relatively large impact of the RGB color channels on the overall performance. Our initial expectation was that only the depth image would contain relevant features for completing the 3D shape. There are two possible explanations for this result. First, the depth image has typically more noise than the RGB image, which may lead to less good feature extraction when only using the depth image. Second, the data augmentations for the RGB color channels were more extensive than for the depth channel, which may have resulted in a better generalization performance when using RGB-D images rather than just depth images. A third interesting outcome is that training without constrastive loss resulted in a lower Chamfer distance and a higher F-score. One explanation for this result is that the contrastive loss may only be beneficial if the images were obtained from different camera perspectives, as was the case in <cit.>. In our experiment, the potato tubers were photographed from the same camera perspective, which resulted in relatively similar 3D shapes, making it more difficult for the contrastive loss function to separate the embedding space. Two other observations from the ablation study that are useful for implementing CoRe++ on an operational harvester are: an instance segmentation algorithm is preferred over an object detection algorithm (as highlighted by the second ablation), and a lighter encoder network may be more favorable for high-throughput shape completion, but it comes at the expense of the accuracy (as highlighted by the third ablation). 



§ DISCUSSION


Of the six latent sizes tested, we observed that a latent size of 32 outperformed the other sizes in terms of 3D shape completion and volumetric estimate. This suggests that a moderate latent size strikes a balance between representational capacity and model complexity, allowing RGB-D images to be encoded more effectively while not overfitting. The latter was clearly demonstrated by the latent space interpolation, which generated the most realistic potato shapes and sizes for latent size 32. Compared to the literature, our observations are similar to those of <cit.>, who found that a latent size of 28 was best for optimizing a convolutional variational autoencoder on spectral topographic maps. Since <cit.> did their research on 25 latent sizes, this suggests that even better results could have been obtained if we had tested more latent sizes. Future work could investigate such a finer-grained latent size analysis or further explore the relationship between neural network architecture and latent size. Such an analysis could potentially provide new insights into how to further optimize the latent space for 3D shape completion tasks. 

With CoRe++ better 3D shape completion results were obtained compared to the linear regression model and the original CoRe implementation of <cit.>. When we compare our results with those of <cit.>, we can conclude that our results on potato are similar (in comparison with strawberry), or better (in comparison with sweet pepper). An important remark is that the obtained results may depend on the average complexity of the shape that has to be completed. Potato has on average a less complex shape than sweet pepper, which makes it easier for the network to learn a generic shape thus enabling a better performance. In future research, we want to test our CoRe++ model on crops or fruits with a more complex shape, such as pineapple, dragon fruit and Romanesco broccoli. We encourage fellow researchers to test our publicly available software on other 3D shapes within the agricultural domain or beyond. 

Our research has provided valuable insights and steps towards the practical application of CoRe++ on a potato harvester. Nevertheless, we think there are potential improvements in software and hardware that could further improve the performance. Regarding software, we would like to explore the use of other 3D shape completion networks, such as the one by <cit.>, who used a transformer network. The obtained results with this new transformer network on sweet pepper and strawberry were significantly better than those of the original CoRe implementation <cit.>. Another advantage of using a transformer network is that it is end-to-end trainable, which is advantageous over our CoRe++ network that needs to be trained in two stages. A hardware improvement that could potentially improve the 3D shape completion is equipping the conveyor with rotating rollers in the area where the camera is placed. The rotating rollers cause the potato tubers to rotate gradually as they pass underneath the camera, meaning that almost the entire shape of the potato can be photographed. This can both simplify and improve the 3D shape completion. In a scenario like this, it would also be useful to test whether the contrastive loss function has an improving effect on completing the 3D shapes of the potato tubers when they rotate. 

To further test the applicability of CoRe++ on a potato harvester, it is important to conduct a more in-depth evaluation on potato tubers of different cultivars. This evaluation will provide a better understanding of the overall generalization performance of CoRe++ at the farm level. In such an analysis, it is also important to conduct the test on potato tubers without the colored thumbtack, as the thumbtack's color and shape can potentially affect the 3D shape completion result. A future evaluation could also benefit from performing the analysis on a more balanced dataset. In our study, potato tubers were randomly selected from the conveyor belt, but this led to an under-representation of the small and large tubers and those with irregular shape. In data-driven approaches such as ours, an unbalanced dataset would usually result in worse results on these minority cases, which is undesirable for a full-field harvest monitoring. Future research should also focus on reducing the degree of human subjectivity in selecting tubers on the conveyor belt by analyzing a batch of potatoes of different sizes at one time rather than manually selecting individual tubers. A final remark is that the performance of CoRe++ on a potato harvester will be affected by the instance segmentation algorithm that provides the necessary binary mask for CoRe++ to complete the 3D shape. Therefore, it is important to also investigate the sensitivity of CoRe++ to the segmentation outputs of the proposed instance segmentation algorithm. 



§ CONCLUSIONS


In this study, we investigated a high-throughput 3D shape completion network for its applicability for estimating the volume of potato tubers on an operational harvester. Our research revealed that latent size 32 had the best 3D shape completion result. With that latent size, our CoRe++ network had an RMSE of 22.6 ml on the volumetric estimate, and this was better than the RMSE of the linear regression (31.1 ml) and the original CoRe network (36.9 ml). We also found that the RMSE of CoRe++ could be further reduced to 18.2 ml when performing the 3D shape completion in the center of the RGB-D image. With an average analysis time of 10 milliseconds per potato tuber, CoRe++ enables a high-throughput 3D shape completion up to 100 potato tubers per second. Hence, we can conclude that our network is able to quickly and accurately estimate the volume of fast-moving potato tubers on an operational harvester. Future improvements lie in expanding the dataset, testing on more potato cultivars, implementing a transformer-based network, and implementing rotating rollers on the harvester's conveyor belt. We encourage other researchers to reuse our dataset and software. 



§ ACKNOWLEDGEMENTS

We would like to thank Okada Farm for providing the potato field on which we acquired the RGB-D images. We thank Ting Jiang, Sylvain Grison, Yuto Imachi and Irena Drofová for their help with the image acquisition and ground truth measurements. 




bib-style






 Paper Title: StreetSurfaceVis: a dataset of crowdsourced street-level imagery with semi-automated annotations of road surface type and quality.




Spin-wave Goos-Hänchen effect induced by 360 degree domain walls in magnetic heterostructures
    Jie Lu
    August 5, 2024
=============================================================================================



§ ABSTRACT 
Road unevenness significantly impacts the safety and comfort of various traffic participants, especially vulnerable road users such as cyclists and wheelchair users. This paper introduces StreetSurfaceVis, a novel dataset comprising 9,122 street-level images collected from a crowdsourcing platform and manually annotated by road surface type and quality. The dataset is intended to train models for comprehensive surface assessments of road networks. 
Existing open datasets are constrained by limited geospatial coverage and camera setups, typically excluding cycleways and footways.
By crafting a heterogeneous dataset, we aim to fill this gap and enable robust models that maintain high accuracy across diverse image sources. 
However, the frequency distribution of road surface types and qualities is highly imbalanced. 
We address the challenge of ensuring sufficient images per class while reducing manual annotation by proposing a sampling strategy that incorporates various external label prediction resources.  
More precisely, we estimate the impact of (1) enriching the image data with OpenStreetMap tags, (2) iterative training and application of a custom surface type classification model, (3) amplifying underrepresented classes through prompt-based classification with GPT-4o or similarity search using image embeddings. We show that utilizing a combination of these strategies effectively reduces manual annotation workload while ensuring sufficient class representation.










§ INTRODUCTION


Road damages have a significant impact on the comfort and safety of all traffic participants and have been identified as a major cause of traffic accidents <cit.>. 
This is particularly true for vulnerable road users such as  cyclists <cit.>, wheelchair users <cit.> and individuals employing inline skates <cit.>, cargo bikes <cit.>, scooters <cit.>, or strollers.
This issue has sparked a large body of research on methods that apply deep learning models to street-level imagery for road surface condition assessment <cit.>.
Yet, road damage may not reflect the full range of factors that influence a driver's experience. For example, the smoothness of sett (regular-shaped cobblestone) is rather determined by the flatness of the utilized stones. 
Rateke et al. <cit.> developed a hierarchical vision-based approach, first predicting the surface type and then employing specific models for each type to classify quality. However, the utilized datasets are too restricted to obtain models that generalize well to new data <cit.>. 
Typical street-level image datasets are commonly collected in good weather conditions, using only a single vehicle and camera setup within a limited geographic boundary, e.g.,  KITTI <cit.>, CaRINA <cit.>, or Oxford RoboCar <cit.>. Cityscapes <cit.> is a dataset of 20,000 images of street scenes recorded in 50 German cities tailored for autonomous driving applications. Perspectives of sidewalks and cycleways are not considered and labels consist of semantic segmentation, while surface type and quality information are unavailable.


We aim to create a street-level image dataset containing a substantial amount of images for each pertinent surface type and quality,

fostering the training of robust classification models. 
We utilize the crowdsourcing platform Mapillary <cit.> to gather images, as these are contributed by individuals from various regions using different devices, camera angles, and modes of transportation, resulting in a heterogeneous dataset.

According to data from the crowdsourcing geographic database OpenStreetMap (OSM), distributions of surface type and quality are highly skewed in Germany: For instance, asphalt is the predominant road type, accounting for 47% of tagged road segments, while only about 3%  are made of sett. Similarly, for 54% of asphalted roads, the quality is considered `good', while only 1% is rated `bad'. Consequently, the difficulty lies in gathering sufficient images for every relevant class without an infeasible manual labeling effort.

This paper introduces StreetSurfaceVis, a new street-level image dataset comprising 9,122 images from the crowdsourcing platform Mapillary, labeled for road surface type and quality by human experts[<https://doi.org/10.5281/zenodo.11449977> The referenced repository contains the dataset, a description, a labeling guide, and a 
datasheet documenting the dataset.

]. Additionally, we present and evaluate different strategies for semi-automated annotation to efficiently amplify underrepresented classes in the dataset. These strategies include pre-filtering using OSM tags, iterative training and application of type classification models, prompt-based image classification with GPT-4 models, and similarity-based search using image embeddings.




§ DATASET CONSTRUCTION





  
Image base. Our dataset is based on images from Mapillary <cit.> limited to the geographical boundaries of Germany. Launched in 2013, the crowdsourcing platform Mapillary provides openly available street-level images. Contributors can use, among others, the Mapillary smartphone app to capture georeferenced image sequences during their trips by, e.g.,  car, bicycle, or on foot. Thus, the dataset encompasses not only images from roadways but also cycleways and footways. 
As of January 2024, Mapillary contains about 170 million images in Germany, with more than 50% captured within the last three years.
The geographic coverage depends on the contributors within each region, resulting in hundreds of thousands of images for every major city in Germany.
As any contributor can upload images, the dataset exhibits varying quality levels, influenced by factors such as the device used, its mounting (e.g., the car dashboard is visible in the frame), or prevailing lighting and weather conditions. Hence, some images are dark, blurred, or contain motifs other than street-level scenes, such as traffic signs, cars, or trees. 



  
Image selection. To increase the dataset's heterogeneity, we restrict the number of images for each location and sequence[During recording, the Mapillary app captures images every few seconds, with all images from a trip belonging to the same sequence.] per class. This reduces the number of images taken by the same person on one trip and thus increases spatial diversity, camera specifications, environmental conditions, and photographic perspectives. Specifically, we limit the number of images per geographic unit (XYZ style spherical mercator tiles on zoom level 14[We thereby adhere to the same geographic unit as utilized by the Mapillary API for computational feasibility.], which is roughly equivalent to ∼ 1.5 x 1.5 km grid cells) to 5 and the number of images per sequence to 10.

For modeling, we recommend using a train-test split where the test data includes geospatially distinct areas, thereby ensuring the model's ability to generalize to unseen regions is tested. We propose five cities varying in population size and from different regions in Germany for testing, comprising 776 images tagged accordingly.




  
Labeling scheme. 
Our labels for surface type and quality primarily align with the OSM road segment tags  and , respectively.  
While  <cit.> describes the surface type such as `asphalt',  <cit.> reflects the physical usability of a road segment for wheeled vehicles, particularly regarding its regularity or flatness <cit.>.
Our labeling scheme includes those classes that are important from a traffic perspective and represent a relevant portion of street types in Germany. This results in the type labels asphalt, concrete, paving stones, sett, and unpaved.
For the quality label, we restrict to five of eight proposed levels, ranging from excellent (suitable for rollerblades), good (racing bikes), intermediate (city bikes and wheelchairs), bad (normal cars with reduced velocity) to very bad (cars with high-clearance). 
The final scheme comprises 18 classes of type and quality combinations, displayed in Table <ref>, as not all quality labels are suitable for all surface types. See Figure <ref> for example images and labels.







  
Manual annotation. After conducting a thorough explorative analysis of Mapillary images, the first three authors developed an annotation guide containing quality level descriptions with example images and underwent self-organized training to manually label surface type and quality. The instructions include labeling the focal road located in the bottom center of the street-level image. In cases where the focus is ambiguous, such as when two parts of the road (e.g., the cycleway and footway) are depicted equally, or when the surface could not be classified due to factors such as snowy roads, blurry images, or non-road images, the image is sorted out.
If the surface quality falls between two categories, annotators are directed to select the lower quality level. Annotators are encouraged to consult each other for a second opinion when uncertain.
For annotation, we use Labelstudio <cit.>, as it allows to preset labels from pre-labeling strategies.

To evaluate inter-rater reliability, 180 heterogeneous images were independently rated by all three annotators.
Twelve images were marked for revision by at least one annotator, and a further 49 were discarded by at least one annotator for the above reasons, all of which were excluded for the calculation of inter-rater reliability.
Krippendorff's α <cit.> for surface type is calculated at 0.96, indicating a high level of agreement. 
Surface quality, treated as an ordinal scale variable, achieves a Krippendorff's α of 0.74.
While this is generally deemed an acceptable level of agreement, it reflects the fluid class transitions of quality. 

An overview of the final annotated dataset is given in Table <ref>.






§ ANNOTATION STRATEGIES


Using the frequency distribution of OSM tags as a baseline estimate[The OSM distribution is likely an overestimation of the frequency of underrepresented classes, as main roads with good quality are typically more frequented, and thus presumably have more images.], manual labeling of randomly sampled images would be highly inefficient. For example,  to obtain 7 images of asphalt-bad, 1,000 images would need to be checked (cf. Table  <ref>). Thus, our goal is to employ pre-selection strategies that yield samples for manual labeling where underrepresented classes have a higher frequency than indicated by the OSM baseline. 
We evaluate three strategies: 
(1) enriching the image dataset with OSM tags; (2) iterative training and application of a model classifying surface type; and (3) amplifying underrepresented type-quality classes prompting with GPT-4o or through a similarity search using image embeddings. 
Table  <ref> presents the improvements through the first two sub-strategies in terms of precision*100, based on 19,747 images processed according to the strategy shown in Figure <ref> (marked in green and yellow). 









  
Pre-labeling via OSM tags


OSM  <cit.> contains  tags for 49%  and  tags for 16% of road segments in Germany, as of January 2024. We incorporate this information by spatially intersecting with geolocations of Mapillary images[For computational feasibility, we refrain from intersecting all Mapillary images and use a sample of Mercantile tiles where the desired classes occur particularly frequently.] and assigning the  and  labels of the closest OSM road segment within a maximum distance of two meters.
To eliminate ambiguous street intersections, we cut off 10% of the start and end of each road segment beforehand.

We test the accuracy of this approach based on a sample of 100 images for each of the 18 classes in terms of OSM labels (Batch 1 in Figure  <ref>). OSM labels and manual annotation agree on 69% of surface type labels, and among these, the quality is correct for 55% of the images. However, the errors are not equally distributed: while 94% of asphalt images are correctly pre-labeled, only 54% of paving stones and sett are correct, both being underrepresented surface types.
This strategy thus increases the precision for each class substantially, albeit with varying impact. For example, to obtain 7 images from the underrepresented class asphalt-bad, around 100 instead of 1,000 images need to be reviewed (cf. Table  <ref>).
We refer to the labels resulting from this strategy as OSM pre-labels.

Incorrect type labels mainly result from mixing up adjacent road parts, as OSM sometimes lacks separate geometries for cycleways and footways. 
Differences in quality labels are likely due to varying subjective assessments by OSM contributors. Additionally, GPS inaccuracies as well as time differences between image capturing and road segment tagging in OSM are plausible sources of discrepancies, for both type and quality.




  
Pseudo-labeling with a type classifier


To further increase pre-selection precision, we iteratively train a classification model to predict the type and use its predictions as pseudo-labels.
Since type is easier to classify than quality, a smaller amount of data should be sufficient to obtain valuable pseudo-labels. 
More precisely, we fine-tune EfficientNetV2-S <cit.>, pre-trained on ImageNet, on the first annotated batch to predict the type.
The model is then applied to the next batch of images. All images where the prediction matches the OSM pre-label are selected for manual annotation; this combination of labels achieves an average precision of 95% for the surface type, with the lowest precision of 89% for paving stones over all batches. To reduce bias towards easy-to-classify examples, a random sample of 10% from the excluded images is manually annotated. 
In the next iteration, the training set is extended with the manually annotated images from the previous round. The batch composition is also adjusted towards underrepresented classes, aiming for 300-400 images per class. 
This procedure leads to significant improvements for certain classes. For example, paving stones-excellent achieves a precision of 30% with a total of 342 images, while  asphalt-good shows an excess of images totaling 1,334. 

However, the precision remains very low for some classes, e.g., less than 4% (cf. Table  <ref>) for paving stones-bad with only 30 images gathered at this point. Thus, continuing this procedure would be infeasible.  




  
Expanding underrepresented classes through prompting and similarity search

We evaluate two approaches to efficiently enlarge classes that remain underrepresented. Our first approach uses prompt-based image classification with OpenAI's GPT-4V <cit.> and GPT-4o <cit.> models, which can generate textual output from image-text input. 
Previous studies have demonstrated the potential of GPT-4V for automated image labeling in various application domains, including 

street intersections <cit.> and traffic scenery <cit.>. 
GPT-4o, released in May 2024, is expected to be similarly effective at half the cost (at the time of release).

As an alternative, we explore  similarity search using image embeddings <cit.> from OpenAI's CLIP  <cit.>, DINOv2 <cit.>, and our fine-tuned EfficientNet-based type classifier. Specifically, annotated images from the class of interest are used as the query, and all images with a cosine similarity score above a certain threshold are pre-labeled as class members. 

For both approaches, we restrict the search space using the previously described strategy combining OSM tags and type classification. Due to cost and time constraints, we limit the experiments to three underrepresented classes. 
Using a validation dataset of randomly sampled manually annotated images (50 for the classes asphalt-bad and paving stones - intermediate and 30 for paving stones - bad), we systematically 
evaluate base models and hyperparameters of both approaches.

Note that in both cases, a higher precision implies a lower estimated effort in human post-annotation, while a higher recall correlates with a larger class increase and, in the prompting scenario, with a lower monetary cost.

Prompting configurations include two different image cropping styles, varying levels of detail in class definitions, zero-shot versus one-shot prediction, and processing a batch versus one image per prompt. While cropping shows only minimal effect, one-shot outperforms the zero-shot setting, albeit with a larger monetary cost, resulting in a similar price per hit. GPT-4o achieves notably better results than GPT-4V for all classes. The best configuration in terms of F1 score and cost-effectiveness consists of GPT-4o, lower half-center cropping, shortened definitions, one-shot prediction, and one request per prompt (see Appendix <ref> for the utilized final prompt).

To decide on the similarity threshold, we compute the optimal ROC cut-off value for each embedding model and class. 
All three embeddings give similar and reasonable results for the two larger classes with optimal cut-off values around 0.85 for CLIP and 0.6 for the other two models. The type-classifier slightly outperforms DINOv2 and CLIP on average and is therefore selected for the following experiment.
The smallest class paving stones - bad remains difficult for all three models, e.g., our embedding returns nearly all images given the optimal threshold.



The models are then applied in their optimal configurations to a dataset compiled as follows: around 20M Mapillary images are used for image selection as described in Section <ref> and pre-labeled using OSM tags and type pseudo-labels. From the remaining data, up to 1,000 images per pre-label are drawn at random. Note that for paving stones-bad only 210 images are available. 
As shown in Table <ref>, both approaches yield a substantial improvement. However, prompting with GPT-4o achieves substantially higher precision than the similarity search, with values between 40% and 65%. 





While the described strategy has shown to have a notable impact (cf. Table <ref>), it substantially limits the search space (for both GPT-4o and the similarity search) due to the sparsity of OSM tags. 

Moreover, the reliance on OSM tags yields a potential selection bias as the distribution can be assumed to depend on factors such as urbanity and OSM community. 
To estimate the efficacy of prompting with GPT-4o  without OSM-based filtering, we apply only the type classification model to a random sample of 20,000 Mapillary images to obtain pseudo-labels and thus define the search space. From the resulting 15,100 images classified as asphalt, we prompt GPT-4o with a random sample of 2,000 images, as well as all 712 images pseudo-labeled as paving stones.
The results show a reduced precision of  18%-31%, thus higher manual labeling effort. 
Nevertheless, there remains a relevant increase to the OSM pre-label baseline, providing a viable method if the OSM tag pre-labeled search space is exhausted.
However, costs substantially rise: with about $0.01 per GPT-4o prompt, obtaining a correctly labeled image without pre-labeling via OSM costs $0.12 to $3.45, depending on the class, compared to $0.07 and $0.18 in the previous experiment.




§ CONCLUSION AND DISCUSSION


We presented StreetSurfaceVis, a new street-level imagery dataset labeled according to surface type and quality. To avoid inefficient manual labeling work, we tested different strategies, in particular, to increase the pre-selection of under-represented classes for further manual annotation. As the most feasible strategy, we identified a combination of intersecting OSM tags, predicting surface type with a classification model, and prompting with GPT-4o. We hope that our results will be useful for other researchers and practitioners to efficiently create similar datasets, e.g., for other regions. 

Generally, performance increases of GPT-4o and similar models are to be expected in the future, further amplifying the viability of this approach. To reduce dependency on OpenAI and monetary cost, future work should evaluate open-source alternatives. 
Similarity search is more efficient in computational and monetary terms, therefore currently remaining a viable alternative despite inferior results. The results varied between classes and showed the worst performance for paving stones - bad, where there was no improvement to the baseline. Further experiments, especially with more images for the smallest class, as well as advancements, could be explored, e.g., by incorporating different clustering strategies <cit.>.


We envision extensions of the dataset to include road-type information as well as unsuitable images, such as blurred ones or those without a road in focus. Including these will enable training models to automatically check image suitability, thus improving classification robustness and enhancing the GPT-4o prompting strategy. 











§ LABELING SCHEME





 §.§ Asphalt



Quality     Description 

excellent     As good as new asphalt, on which a skateboard or rollerblades will have no problem. 

good     Asphalt showing the first signs of wear, such as narrow, smaller than 1.5 cm cracks, or wider cracks filled up with tar, shallow dents in which rainwater may collect, which may cause trouble for rollerblades but not for racing bikes. 

intermediate     Asphalt roads that show signs of maintenance, such as patches of repaired surface, wider cracks larger than 2cm. Asphalt sidewalks may contain potholes, but these are small, shallow (<3cm deep) and can be easily avoided. Asphalt driving lanes show damage due to subsidence (depressions of a scale >50 cm) or heavy traffic (shallow ruts in asphalt caused by trucks in summer). This means that the road can be used by normal city bikes, wheelchairs and sports cars, but not by a racing bike. 

bad     Damaged asphalt roads that show clear signs of maintenance: This might include potholes, some of them quite deep, which might decrease the average speed of cars. However, it isn’t so rough that ground clearance becomes a problem. Meaning that the street causes trouble to normal city bikes but not a trekking bike and a car. 





 §.§ Concrete



Quality     Description 

excellent     As good as new concrete, on which a skateboard or rollerblades will have no problem. 

good     Concrete road showing the first signs of wear, such as narrow, smaller than 1.5 cm cracks, or wider cracks filled up with tar, shallow dents in which rainwater may collect, which may cause trouble for rollerblades but not for racing bikes. 

intermediate     Concrete roads that show signs of maintenance, such as patches of repaired surface, wider cracks larger than 2cm. Concrete sidewalks may contain potholes, but these are small, shallow (<3cm deep) and can be easily avoided. Concrete driving lanes show damage due to subsidence (depressions of a scale >50 cm) or heavy traffic (shallow ruts in concrete caused by trucks in summer). This means that the road can be used by normal city bikes, wheelchairs and sports cars, but not by a racing bike. 

bad     Heavily damaged concrete roads that badly need maintenance: many potholes, some of them quite deep. The average speed of cars is less than 50% of what it would be on a smooth road. However, it isn’t so rough that ground clearance becomes a problem. Meaning that the street causes trouble to normal city bikes but not a trekking bike and a car. 





 §.§ Paving stones



Quality     Description 

excellent     Newly installed and regularly laid paving stones that show no signs of wear. Gaps may be visible, but they are small and uniform and do not significantly affect the driving experience. 

good     Paving stones showing first signs of wear or newly installed stones with visible but uniform gaps between them. While still suitable for most activities, these surfaces may pose minor challenges for rollerblades and skateboards but remain navigable for racing bikes. 

intermediate     Characterized by paving stones exhibiting multiple signs of wear, such as shifted heights, potholes, or cracks. This grade allows for the comfortable passage of normal city bikes and standard vehicles but may prove challenging for racing bikes. 

bad     Heavily uneven or damaged paving stones in dire need of maintenance, featuring significant height disparities and numerous deep potholes. While ground clearance remains sufficient for most vehicles, the surface severely impedes travel, particularly for standard city bikes. 





 §.§ Sett



Quality     Description 

good     The best sett roads with flattened stones and gaps that are at most small or filled up. The surface might cause trouble for rollerblades but not for racing bikes. 

intermediate     The surfaces of the sett stones are not completely flat, or there may be slightly larger gaps between the stones, this causes problems for racing bikes and slows down city bikes and cars. 

bad     Sett stones with large and possibly uneven gaps or uneven stone surfaces or damaged stones, resulting in an overall bumpy surface: This results in a highly uncomfortable driving experience for city bikes. The average speed of cars is less than 50% of what it would be on a smooth road. However, it isn’t so rough that ground clearance becomes a problem. 





 §.§ Unpaved



Quality     Description 

intermediate     The best unpaved roads that have a compacted surface. This grade allows for the passage of normal city bikes and standard vehicles but may prove challenging for racing bikes. 

bad     Unpaved roads that do not have a smooth and compacted surface but ones that can still be used with a car or trekking bike but not with a city bike. This category also includes hiking paths, which are too narrow for cars. 

very_bad     Unpaved roads with potholes, ruts or generally a highly uneven surface not safely passable with a regular passenger car but still passable with an average SUV with higher ground clearance. This category also includes highly uneven hiking paths, which are too narrow for cars. 





§ GPT 4-O PROMPT



The following shows the utilized prompt for one-shot asphalt image prediction using GPT 4-o. Prompts for other surface types are adjusted according to type-specific labeling schemes (see Appendix  <ref>).





 Paper Title: Fine-grained Metrics for Point Cloud Semantic Segmentation.













Z.Lu et al.





School of Automation, Nanjing University of Science and Technology, Nanjing 210094, China
 
 su@njust.edu.cn

School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing 210094, China




Fine-grained Metrics for Point Cloud Semantic Segmentation
    Zhuheng Lu1Ting Wu1 Yuewei Dai1 Weiqing Li2 Zhiyong Su1()
    July 2024
=============================================================



§ ABSTRACT 

Two forms of imbalances are commonly observed in point cloud semantic segmentation datasets: (1) category imbalances, where certain objects are more prevalent than others; and (2) size imbalances, where certain objects occupy more points than others.
Because of this, the majority of categories and large objects are favored in the existing evaluation metrics.
This paper suggests fine-grained mIoU and mAcc for a more thorough assessment of point cloud segmentation algorithms in order to address these issues.
Richer statistical information is provided for models and datasets by these fine-grained metrics, which also lessen the bias of current semantic segmentation metrics towards large objects.
The proposed metrics are used to train and assess various semantic segmentation algorithms on three distinct indoor and outdoor semantic segmentation datasets.







§ INTRODUCTION

Point cloud semantic segmentation algorithms aim to classify the point clouds at the point level. 
Algorithm optimization and improvement heavily depend on the choice of suitable and thorough evaluation metrics for various algorithms and application circumstances.
In the past few years, the overall accuracy (OA) is provided to evaluate the segmentation results. 
However, it is inappropriate to use only OA when the semantic segmentation datasets have long-tailed distributions, since it is biased to categories with larger number of points.
As a result, mean class accuracy (mAcc) is then used in the semantic segmentation experiment, which computes and averages the point accuracy of each category based on the datasets.
Besides, mean intersection over union (mIoU) is suggested as an evaluation metric of segmentation results, nevertheless, as the mAcc does not take false positives into account. 
Since then, dataset-level mAcc and mIoU is widely used for semantic segmentation tasks on different datasets, such as indoor scene dataset ScanNet <cit.>, S3DIS <cit.> and outdoor scene dataset Semantic3D <cit.>.

Among the existing dataset-level evaluation metrics, the prediction results of true positive, false positive, true negative and false negative are counted across all points over the whole dataset. 
These evaluation metrics still face several problems:


  * Existing evaluation metrics exhibit a bias in favor of larger objects due to the imbalances. 
Nonetheless, a significant proportion of points are unequally distributed among categories in the majority of point cloud semantic segmentation datasets. 


  * Existing dataset-level evaluation metrics fail to capture valuable statistics about segmentation algorithms on a single point cloud or object instance, preventing comprehensive comparisons.




In this paper, fine-grained mIoU and mAcc are proposed for the evaluation of semantic segmentation algorithms.
Specifically, the point cloud-level metrics are first calculated to deal with the category imbalances. 
Then, the approximate instance-level metrics are further calculated to cope with the size imbalances.
These fine-grained metrics reduce bias toward large objects.
Furthermore, they supply a wealth of statistical information, allowing for more robust and comprehensive comparisons.



§ RELATED WORK



In the semantic segmentation task, each point in the point cloud is assigned a predictive semantic label by the algorithm.
The predicted labels are compared with the manually labeled labels, and four indicators are counted for each category: true positive, false positive, false negative, and true negative. 
The most common evaluation metrics include: OA, mAcc, and mIoU.













OA represents the proportion of correctly segmented points to the total number of points <cit.>.
However, overall accuracy is biased to categories with large number of points.
Moreover, the overall accuracy contains limited information, making it easy to overlook the phenomenon of poor segmentation performance of a certain category.
It cannot accurately report the segmentation results of particular categories.

mAcc is introduced to average point-wise accuracy across all categories <cit.>.
It is applicable to datasets with a large and imbalanced number of categories, and can represent the semantic segmentation accuracy of each category.
However, mAcc does not take false positives in account, which leads to over-segmentation of the results.

IoU, also known as the Jaccard Similarity Coefficient (JSC), is now the most commonly employed in semantic segmentation <cit.>.
In point cloud semantic segmentation, IoU represents the overlap rate between the predicted mask and the labeled point cloud, which is used to evaluate the accuracy of the predicted segmentation area. 
It is also used in object detection and instance segmentation.
Mean intersection over union (mIoU) is the arithmetic mean of the IoU values of all categories, used to evaluate point overlap in the overall dataset.
As the most widely used metric in point cloud semantic segmentation, the mIoU indicates the prediction ability of the segmentation methods to the correct point in the datasets.

Therefore, selecting appropriate and comprehensive evaluation metrics for different algorithms plays an important role in algorithm optimization and improvement. 
Traditional dataset-level evaluation metrics cannot capture valuable statistical information regarding the performance of segmentation methods on a single point cloud or instance, preventing comprehensive comparisons.



§ FINE-GRAINED MIOU



This section first reviews the dataset-level mIoU^D, and then proposes point cloud-level mIoU^P, mIoU^C and instance-level mIoU^I.



 §.§ Dataset-level mIoU^D



The dataset-level mIoU^D is calculated by accumulating the segmentation results of all points in the entire dataset. 
For each category, the IoU_c^D is defined as:

    IoU_c^D=∑_p=1^P TP_p,c/∑_p=1^P( TP_p,c+ FP_p,c+ FN_p,c),

where P is the number of point clouds, TP_p,c, FP_p,c and FN_p,c respectively represent the number of TP, FP, and FN points in the c-th category of p-th point cloud. 
Then, dataset-level mIoU^D is defined as:

    mIoU^D=1/C∑_c=1^C IoU_c^D,

where C is the number of categories in the dataset.



 §.§ Point Cloud-level mIoU^P and mIoU^C

The point cloud-level metrics aim to give more detailed assessment for semantic segmentation methods to cope with the category imbalances. 
Since not all categories appear in different point cloud models, there will be statistical bias in the statistical results of the dataset-level evaluation metrics.
Therefore, the NULL representation is introduced into the point cloud-level mIoU^P and mIoU^C.
For the p-th point cloud in the c-th category, the IoU_p,c is defined as:

    IoU_p,c= TP_p,c/ TP_p,c+ FP_p,c+ FN_p,c.


Then, based on the IoU_p,c of all categories, the IoU_p^ P of p-th point cloud is defined as follows:

    IoU_p^ P=∑_c=1^C1{ IoU_p,c≠𝐍𝐔𝐋𝐋} IoU_p,c/∑_c=1^C1{ IoU_p,c≠𝐍𝐔𝐋𝐋},

where 1 is the indicator function, and only non-empty categories are counted. 
If a category does not appear in the point cloud, it is marked as NULL.

Finally, IoU_p^ P is obtained by calculating the average value of each point cloud-level IoU_p^ P:

    mIoU^P=1/P∑_p=1^P IoU_p^ P.





The results of calculating the average by category and then by point cloud are different from those of calculating the average by point cloud and then by category due to the NULL values, as shown in Figure <ref>. One drawback of the mIoU^P metric is that it tends to favor category C_1, which appears more frequently in point clouds. 
To address this bias, the mIoU^C metric is first counted by point cloud and then by category. 
The point cloud-level IoU_c^ C of a certain category is defined as:

    IoU_c^ C=∑_p=1^P1{ IoU_p,c≠𝐍𝐔𝐋𝐋} IoU_p,c/∑_p=1^P1{ IoU_p,c≠𝐍𝐔𝐋𝐋}.



Similar to mIoU^P, averaging these point cloud-level metrics can yield mIoU^C:

    mIoU^C=1/C∑_c=1^C IoU_c^ C.




 §.§ Instance-level mIoU^I

The instance-level mIoU^I provides a more precise evaluation of segmentation algorithms to deal with size-variance. 
Specifically, for the i-th instances of c-th category in p-th point cloud, both TP and FN can be accurately calculated. 
The instance-level FP is approximated by allocating point cloud-level FP proportionally to the size of each instance <cit.>.
Then, the IoU_p,c,i of a a certain instance is defined as:

    IoU_p,c,i= TP_p,c,i/( TP_p,c,i+ FN_p,c,i+S_p,c,i/∑_i=1^I_p,cS_p,c,i FP_p,c),

where FP_p,c is the number of points of the c-th category in the p-th point cloud where the segmentation result is FP, and I_p,c represents the total number of instances belonging to the c-th category in the p-th point cloud. 
S_p,c,i is the total number of points of the i-th instance, which can be defined as:

    S_p,c,i= TP_p,c,i+ FN_p,c,i.


Based on the IoU_p,c,i of each instance, the IoU_c^ I of a certain category can be defined as follows:

    IoU_c^ I=∑_p=1^P∑_i=1^I_p,c IoU_p,c,i/∑_p=1^PI_p,c.


By averaging IoU_c^ I of all categories, the instance-level mIoU^I metric can be obtained as follows:

    mIoU^I=1/C∑_c=1^C IoU_c^ I.





§ FINE-GRAINED MACC



This section first reviews the dataset-level mAcc^D, and then presents point cloud-level mAcc^P, mAcc^C and instance-level mAcc^I.



 §.§ Dataset-level mAcc^D

mAcc^D reports the average segmentation accuracy of various categories in the dataset. 
For the c-th category, the Acc_c^D is defined as:

    Acc_c^D=∑_p=1^P( TP_p,c+ TN_p,c)/∑_p=1^P( TP_p,c+ FP_p,c+ TN_p,c+ FN_p,c).


Then, dataset-level mAcc^D is defined as:

    mAcc^D=1/C∑_c=1^C Acc_c^D.





 §.§ Point Cloud-level mAcc^P and mAcc^C

The point cloud-level mAcc^P and mAcc^C are designed to provide a more comprehensive evaluation of segmentation algorithms with a focus on category imbalance.
The NULL representation is also introduced into the point cloud-level mAcc^P and mAcc^C similar to the point cloud-level mIoU^P and mIoU^C.
For the p-th point cloud in the c-th category, the Acc_p,c is defined as:

    Acc_p,c= TP_p,c+ TN_p,c/ TP_p,c+ FP_p,c+ FN_p,c+ TN_p,c.


The Acc_p^ P for p-th point cloud can be calculated using the Acc_p,c of all categories, which is specified as:

    Acc_p^ P=∑_c=1^C1{ Acc_p,c≠𝐍𝐔𝐋𝐋} Acc_p,c/∑_c=1^C1{ Acc_p,c≠𝐍𝐔𝐋𝐋},

where 1 is the indicator function. Only non-empty categories are counted in the Acc_p^ P. 
Then, the point cloud-level mAcc^P is defined as:

    mAcc^P=1/P∑_p=1^P Acc_p^ P.


Then, segmentation results are counted by point cloud and then by category. 
Specifically, the point cloud level Acc_c^ C of a certain category is defined as:

    Acc_c^ C=∑_p=1^P1{ Acc_p,c≠𝐍𝐔𝐋𝐋} Acc_p,c/∑_p=1^P1{ Acc_p,c≠𝐍𝐔𝐋𝐋}.


The point cloud-level mAcc^C is achieved by averaging all the Acc_c^ C:

    mAcc^C=1/C∑_c=1^C Acc_c^ C.




 §.§  Instance-level mAcc^I

The instance-level mAcc^I is designed to provide a more granular evaluation for segmentation algorithms to deal with size differences among objects.
For the i-th instances of c-th category in p-th point cloud, the Acc_p,c,i is defined as:

    Acc_p,c,i= TP_p,c,i+ TN_p,c,i/( TP_p,c,i+ FN_p,c,i+ TN_p,c,i+S_p,c,i/∑_i=1^I_p,cS_p,c,i FP_p,c).



Then, the Acc_c^ I of a certain category can be defined as follows:

    Acc_c^ I=∑_p=1^P∑_i=1^I_p,c Acc_p,c,i/∑_p=1^PI_p,c.


Finall, the instance-level mAcc^I is defined as:

    mAcc^I=1/C∑_c=1^C Acc_c^ I.




§ EXPERIMENTAL RESULTS


The proposed fine-grained metrics are mainly evaluated on two large-scale indoor datasets and one challenging outdoor dataset, and the benchmark results are reported for each dataset.



 §.§ Datasets


ScanNet. ScanNet <cit.> is a large 3D dataset containing 1,513 point clouds of scans from 707 unique indoor scenes. ScanNet contains a variety of spaces such as offices, apartments, and bathrooms. The annotation of the point clouds corresponds to 20 semantic categories plus one for the unannotated space. We adopt the official train-val split, where there are 1205 training scenes and 312 validation scenes.

S3DIS. S3DIS <cit.> is produced for indoor scene understanding and is widely used in the point cloud semantic segmentation task. S3DIS consists of 3D RGB point clouds of six floors from three different buildings split into individual rooms. Each room is scanned with RGBD sensors and is represented by a point cloud with coordinate information and RGB value. 
It contains 3D scans of 271 rooms with 13 categories, where area 5 is used for testing and the rest for training.  

Semantic3D. Semantic3D <cit.> is a classic outdoor dataset containing urban and rural scenes. The annotation of the point clouds corresponds to 8 semantic categories, including man-made terrain, natural terrain, high vegetation, low vegetation, buildings, hard scape, scanning artifacts and cars. Semantic3D consists of 15 training scans and 15 test scans.










 §.§ Evaluations on mIoU


This section calculates the semantic segmentation results of dataset-based metric mIoU^D, point cloud-level metrics mIoU^P and mIoU^C, and instance-level metric mIoU^I on three datasets. 
The evaluation results are shown in Table <ref>, <ref>, and <ref>, respectively. 
On the ScanNet and Semantic3D datasets, none of the methods performed optimally on all of the evaluation metrics. 
Thus it is necessary to evaluate multiple evaluation metrics at the same time in order to conduct a comprehensive evaluation.
On the S3DIS dataset, the PointMeta <cit.> outperforms other methods on all metrics.
The segmentation results on the point cloud-level mIoU^C and the instance-level mIoU^I are close on the three datasets.
Therefore, when the dataset has no instance-level labels, the segmentation result of the point cloud-level mIoU^C can be utilized as a reference.







In addition, this section contrasts the rank of mIoU^C with the other three metrics across three datasets, as shown in Figure <ref>, <ref> and <ref>, respectively.
It is clear that when the performance of multiple methods is close, the experimental results show local ranking discrepancies on several metrics.
Therefore, in order to obtain a more comprehensive evaluation of different segmentation methods, it is required to compare segmentation metrics of varying granularity.













 §.§ Evaluations on mAcc


This section reports the semantic segmentation results of dataset-level metric mAcc^D, point cloud-level metrics mAcc^P and mAcc^C, and instance-level metric mAcc^I on three datasets, as shown in Table <ref>, <ref> and <ref>, respectively.
No method achieves the best result across all metrics on the ScanNet and Semantic3D datasets.
On the S3DIS dataset, the PointMeta <cit.> model outperforms other methods on all metrics.
Therefore, it is necessary to simultaneously refer to multiple evaluation metrics of different granularity to carry out comprehensive evaluation.







This section also compares the rank of mAcc^C to the other three metrics across three datasets, as illustrated in Figures <ref>, <ref>, and <ref>, respectively.
Figure <ref> shows that different methods have consistent rankings between the mAcc^P and mAcc^C on Semantic3D dataset.
In addition, the segmentation results exhibit local ranking disparities across different metrics when the segmentation performance of different methods is similar.
As a result, the evaluation metrics with different granularity can provide a more comprehensive evaluation.















§ CONCLUSION


This paper proposes fine-grained evaluation metrics of semantic segmentation to address the issue that the current evaluation metrics in point cloud semantic segmentation are biased to most categories and large objects.
The suggested fine-grained mIoU and mAcc evaluation metrics at the point cloud and instance levels provide a more thorough evaluation of the segmentation algorithms.
Various semantic segmentation models are trained and assessed using the suggested metrics on three distinct indoor and outdoor semantic segmentation datasets.
Experiments demonstrate that these fine-grained metrics offer greater statistical information and lessen the bias towards large objects.




§ ACKNOWLEDGMENT


The authors sincerely acknowledge the anonymous reviewers for their insights and comments to further improve the quality of the manuscript.
They also would like to thank the participants in the study for their valuable time.








 splncs04
 
























 Paper Title: Dynamic Gesture Recognition in Ultra-Range Distance for Effective Human-Robot Interaction.








Dynamic Gesture Recognition in Ultra-Range Distance for Effective Human-Robot Interaction





    1st Eran Bamani Beeri
School of Mechanical Engineering 

Tel-Aviv University

Tel-Aviv, Israel 6997801 

eranbamani@mail.tau.ac.il
2nd Eden Nissinman
School of Mechanical Engineering 

Tel-Aviv University

Tel-Aviv, Israel 6997801 

edennissinman@gmail.com
3rd Avishai Sintov
School of Mechanical Engineering 

Tel-Aviv University

Tel-Aviv, Israel 6997801 

sintov1@tauex.tau.ac.il

    August 5, 2024
===================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
This paper presents a novel approach for ultra-range gesture recognition, addressing Human-Robot Interaction (HRI) challenges over extended distances. By leveraging human gestures in video data, we propose the Temporal-Spatiotemporal Fusion Network (TSFN) model that surpasses the limitations of current methods, enabling robots to understand gestures from long distances. With applications in service robots, search and rescue operations, and drone-based interactions, our approach enhances HRI in expansive environments. Experimental validation demonstrates significant advancements in gesture recognition accuracy, particularly in prolonged gesture sequences.



Human-Robot Collaboration, Human-Robot Interaction, Video Recognition, Ultra-Range Gesture Recognition






§ INTRODUCTION

In recent years, Gesture Recognition (GR) in Human-Robot Interaction (HRI) has been advanced significantly. While research primarily focuses on hand gesture recognition <cit.> using conventional cameras and Kinect devices <cit.>, a critical gap remains in recognizing gestures over distances beyond seven meters <cit.>. Bridging this gap is crucial for seamless collaboration between humans and robots, especially in scenarios like surveillance, drone operations, and search and rescue missions.




Ultra-range gesture recognition (URGR) aims to extend recognition capabilities to 25 meters <cit.>, yet considers only static images as the majority of gesture recognition approaches. However, some gestures are inherently dynamic such as waving and beckoning. Additionally, some dynamic gestures may appear as different if only a single frame is used for inference. For instance, a snapshot of a waving gesture may look like a stop gesture. In addition, URGR often faces challenges like image degradation and noise. Video data offers promise, capturing motion dynamics better <cit.>. Deep learning shows potential for recognizing gestures in videos <cit.>, aiding robots in understanding gestures across distances.

In this paper, we introduce the Temporal-Spatiotemporal Fusion Network (TSFN) for URGR in HRI which is effective within a range of 28 meters and operates in real-time. The proposed TSFN model uses Temporal Convolutional Networks (TCN) <cit.> for temporal dependencies and R(2+1)D 
convolutional networks <cit.> for spatiotemporal feature extraction, advancing ultra-range human-robot interaction.




§ METHODS



 §.§ Problem Formulation and Data Collection


GR across diverse environments and distances is essential for effective human-robot communication. Let V_i ∈ℝ^T × H × W × 3 denote video sequences up to 28 meters, with T, H, and W representing frame quantity, height and width, respectively. GR in a video faces challenges due to temporal dynamics, occlusions, and lighting variations. Given a temporal frame sequence V = {v_1, v_2, ..., v_T} and a set of gestures G, the problem entails finding the function F: V → G mapping video sequences to human gestures.

A diverse dataset 𝒟 = {(X_i, y_i, d_i)}_i=1^N is crucial for accurate gesture recognition. It comprises input video X_i, gesture label y_i, and distance d_i ∈ [4, 28] meters from the camera. Webcam-collected data offers varied scenarios, improving model robustness. Challenges include pose/environmental diversity and image quality degradation with distance, leading to reduced resolution and motion blur. Annotations include ground truth gesture labels and distance, aiding model training/evaluation.




 §.§ Models


Our TSFN model combines TCN and R(2+1)D models to create a robust  GR system from video sequences. TCN captures temporal dependencies and the output of a TCN layer is given by



    Y_TCN = σ(W_TCN∗ V + b_TCN),

where W_TCN and b_TCN are the weights and biases of the TCN layer, the operator ∗ denotes convolution, and σ is the Sigmoid activation function.
Video data necessitates 3D convolutions.
A standard 3D convolution applies a filter W ∈ℝ^K × d × k × k, where K is the number of output channels, d is the temporal depth, and k × k is the spatial dimension, given by:

    Y_t,h,w^k = ∑_c=1^C∑_i=1^d∑_j=1^k∑_m=1^k W_c,i,j,m^k · V_c, t+i, h+j, w+m.

In the TCN model, the 3D convolution is factorized into 1D temporal convolution. R(2+1)D networks extend two-dimensional convolutional operations into the temporal domain, separating spatial and temporal convolutions. The R(2+1)D convolution is expressed as

    Y_R(2+1)D = σ(W_spatial∗ (σ(W_temporal∗ V + b_temporal)) + b_spatial),

where W_spatial and W_temporal are the spatial and temporal convolutional weights, respectively. Finally. the outputs from the TCN and R(2+1)D layers are concatenated to form combined features



    F_combined = [Y_TCN, Y_R(2+1)D].

These features are then passed through three fully connected layers to produce the final gesture classification:



    O = σ(W_fc· F_combined + b_fc),

where W_fc and b_fc are the weights and biases of the fully connected layer. This fusion leverages the strengths of both models to enhance GR accuracy.

To train our TSFN model to address the challenges of recognizing human gestures in a video, particularly when the subject is far from the camera, we propose a novel loss function that takes into account the degradation and other phenomena affecting the results. This novel loss function will be designed to balance classification accuracy with robustness to distance-related degradation and is given by

    ℒ = ℒ_CE + α·ℒ_global + β·ℒ_dist + γ·ℒ_robust
,
where ℒ_CE is the cross-entropy loss and ℒ_global is the global context loss that ensures the global context of the video is captured accurately. Loss
ℒ_dist is the distance aware loss that takes into account the degradation effect due to varying distances given by 



    ℒ_dist = 1/N∑_i=1^N d_i ·ℒ_CE(v_i, y_i).

ℒ_dist penalizes more easily for errors when the subject is farther from the camera. Robustness loss ℒ_robust enhances the model's robustness to various degradations by adding a regularization component that minimizes the variance in predictions due to changes in distance:

    ℒ_robust = 1/N∑_i=1^N∑_j=1^M (ℒ_CE(v_i^(j), y_i) - ℒ̅_CE(y_i))^2.

Component ℒ̅_CE is the mean cross-entropy loss for the true label, α, β, γ are hyper-parameters that control the relative importance of each loss component.




§ EXPERIMENTAL RESULTS


To validate the effectiveness of the proposed TSFN model, we conducted experiments comparing its performance with state-of-the-art video recognition models. Figure <ref> illustrates the six gesture categories we focused on: pointing, thumbs-up, thumbs-down, beckoning, stop, and null gestures. The models were tested on a dataset of human gestures observed from 4 to 28 meters. We collected 15 samples of 4-second videos per meter, totaling 2,250 videos for each gesture, with 150 videos used as the test set. Key performance metrics including accuracy, loss, and Mean Average Precision (mAP) are shown in Table <ref>. Figure <ref> shows the accuracy of the TSFN in GR with respect to different distances. The results clearly show the superiority of TSFN over existing methods.







§ CONCLUSION

TSFN redefines ultra-range gesture recognition, which is crucial for seamless Human-Robot Collaboration. By fusing TCNs and R(2+1)D networks, TSFN excels in understanding gestures over vast distances. Future work includes refining TSFN for real-time applications and exploring its potential in dynamic environments including underwater.

99

ahmedhand
S. Ahmed, K. D. Kallu, S. Ahmed, and S. H. Cho, "Hand gestures recognition using radar sensors for human-computer-interaction: A review," Remote Sensing, vol. 13, no. 3, p. 527, 2021.

biswas2011gesture
K. K. Biswas and S. K. Basu, "Gesture recognition using Microsoft Kinect®," in Proceedings of the 5th International Conference on Automation, Robotics, and Applications, pp. 100–103, IEEE, 2011.


nickel2007visual
K. Nickel and R. Stiefelhagen, "Visual recognition of pointing gestures for human-robot interaction," Image and Vision Computing, vol. 25, no. 12, pp. 1875–1884, 2007.

wachs2011vision
J. P. Wachs, M. Kölsch, H. Stern, and Y. Edan, "Vision-based hand-gesture applications," Communications of the ACM, vol. 54, no. 2, pp. 60–71, 2011.

bamani2024ultra
E. Bamani, E. Nissinman, I. Meir, L. Koenigsberg, and A. Sintov, "Ultra-Range Gesture Recognition using a web-camera in Human–Robot Interaction," Engineering Applications of Artificial Intelligence, vol. 132, p. 108443, 2024.

materzynska2019jester
J. Materzynska, G. Berger, I. Bax, and R. Memisevic, "The jester dataset: A large-scale video dataset of human gestures," in Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 2019, pp. 0–0.

lui2012human
Y. M. Lui, "Human gesture recognition on product manifolds," The Journal of Machine Learning Research, vol. 13, no. 1, pp. 3297–3321, 2012.

yang2018temporal
K. Yang, R. Li, P. Qiao, Q. Wang, D. Li, and Y. Dou, "Temporal pyramid relation network for video-based gesture recognition," in 2018 25th IEEE International Conference on Image Processing (ICIP), 2018, pp. 3104–3108.

bu2020human
X. Bu, "Human motion gesture recognition algorithm in video based on convolutional neural features of training images," IEEE Access, vol. 8, pp. 160025–160039, 2020.

tran2018closer
D. Tran et al., "A closer look at spatiotemporal convolutions for action recognition," in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.

lea2016temporal
C. Lea et al., "Temporal convolutional networks: A unified approach to action segmentation," in Computer Vision–ECCV 2016 Workshops: Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part III, vol. 14, pp. 47-54, Springer International Publishing, 2016.

chen2023cnn
J. Chen, J. Wang, Q. Yuan, and Z. Yang, "CNN-LSTM model for recognizing video-recorded actions performed in a traditional Chinese exercise," IEEE Journal of Translational Engineering in Health and Medicine, 2023.

arnab2021vivit
A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lučić, and C. Schmid, "ViViT: A video vision transformer," in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6836–6846, 2021.






 Paper Title: XHand: Real-time Expressive Hand Avatar.








Journal of  Class Files, Vol. 14, No. 8, August 2021
Shell et al.: A Sample Article Using IEEEtran.cls for IEEE Journals





XHand: Real-time Expressive Hand Avatar
    Qijun Gan, Zijie Zhou and Jianke Zhu, Senior Member, IEEE
        
Qijun Gan, Zijie Zhou and Jianke Zhu are with the College of Computer Science and Technology, Zhejiang University, Zheda Rd 38th, Hangzhou, China. Email: {ganqijun, zjzhou, jkzhu}@zju.edu.cn;
Jianke Zhu is the Corresponding Author.
    
==============================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Hand avatars play a pivotal role in a wide array of digital interfaces, enhancing user immersion and facilitating natural interaction within virtual environments. While previous studies have focused on photo-realistic hand rendering, little attention has been paid to reconstruct the hand geometry with fine details, which is essential to rendering quality. In the realms of extended reality and gaming, on-the-fly rendering becomes imperative. To this end, we introduce an expressive hand avatar, named XHand, that is designed to comprehensively generate hand shape, appearance, and deformations in real-time. To obtain fine-grained hand meshes, we make use of three feature embedding modules to predict hand deformation displacements, albedo, and linear blending skinning weights, respectively. To achieve photo-realistic hand rendering on fine-grained meshes, our method employs a mesh-based neural renderer by leveraging mesh topological consistency and latent codes from embedding modules. During training, a part-aware Laplace smoothing strategy is proposed by incorporating the distinct levels of regularization to effectively maintain the necessary details and eliminate the undesired artifacts. The experimental evaluations on InterHand2.6M and DeepHandMesh datasets demonstrate the efficacy of XHand, which is able to recover high-fidelity geometry and texture for hand animations across diverse poses in real-time. To reproduce our results, we will make the full implementation publicly available at <https://github.com/agnJason/XHand>.



3D hand reconstruction, animatable avatar, MANO.


    



§ INTRODUCTION



HAND avatars are crucial in various digital environments, including virtual reality, digital entertainment, and human-computer interaction <cit.>. Accurate representation and lifelike motion of hand avatars are essential to deliver an authentic and engaging user experience. Due to the complexity of hand muscles and the personalized nature, it is challenging to obtain the fine-grained hand representation <cit.>, which directly affect the user experience in virtual spaces.



Parametric model-based methods <cit.> have succeeded in modeling digital human, which offer the structured frameworks to efficiently analyze and manipulate the shapes and poses of human bodies and hands. These models have played a crucial role in various applications, enabling computer animation and hand-object interaction <cit.>. Since they predominantly rely on mesh-based representations, it restricts them to fixed topology and limited resolution of the 3D mesh. Consequently, it is difficult for these models to accurately represent the intricate details, such as muscle, garments and hair. This hinders them from rendering high fidelity images <cit.>. 
Model-free methods offer effective solutions for representing hand meshes through various techniques. Graph Convolutional Network (GCN)-based and UV-based representations of 3D hand meshes <cit.> enable the reconstruction of diverse hand poses with detailed deformations. Lightweight auto-encoders <cit.> further enhance real-time hand mesh prediction. Despite these advancements in capturing accurate hand poses, these methods still fall short in preserving intricate geometric details.

Recently, neural implicit representations <cit.> have emerged as powerful tools in synthesizing novel views for static scenes. Some studies <cit.> have expanded these methods into the realm of articulated objects, notably the human body, to facilitate photo-realistic rendering. LiveHand <cit.> achieves real-time rendering through a neural implicit representation along with a super-resolution renderer. Karunratanakul et al. <cit.> present a self-shadowing hand renderer. Corona et al. <cit.> introduce a neural model LISA that predicts the color and the signed distance with respect to each hand bone independently. Despite the promising results, it struggles to capture intricate high-frequency details and lacks capability of real-time rendering. Meanwhile, Chen et al. <cit.> make use of occupancy and illumination fields to obtain hand geometry, while the generated hand geometry lacks the intricate details and appears to be smoothing surface. These methods have difficulties in recovering the detailed geometry that usually plays a crucial role in photo-realistic rendering. 

In addition to hand modeling methods, several studies have focused on reconstructing animatable human bodies or animals <cit.>. Building accurate human body models presents significant challenges due to the complex deformations involved, particularly in capturing fine details such as textures and scan-like appearances, especially in smaller areas like hands and faces <cit.>. To address these challenges, several approaches have been developed with detailed 3D scans. For instance, previous works <cit.> have focused on establishing correspondences between pose space and standard space through techniques such as linear blend skinning and inverse skinning weights. These advancements collectively contribute to more precise and realistic human body modeling, while their results for hand modeling remain smooth.

To address these challenges, we propose XHand, an expressive hand avatar that achieves real-time performance (see Fig. <ref>). Our approach includes feature embedding modules that predict hand deformation displacements, vertex albedo, and linear blending skinning (LBS) weights using a subdivided MANO model <cit.>. These modules utilize average features of the hand mesh and compute feature offsets for different poses, addressing the difficulty in directly learning dynamic personalized hand color and texture due to significant pose-dependent variations. By distinguishing between average and pose-dependent features, our modules simplify the training task and improve result accuracy. Additionally, we incorporate a part-aware Laplace smoothing term to enhance the efficiency of geometric information extraction from images, applying various levels of regularization.

To achieve photo-realistic hand rendering, we use a mesh-based neural renderer that leverages latent codes from the feature embedding modules, maintaining topological consistency. This method preserves detailed features and minimizes artifacts through various regularization levels. We evaluate our approach using the InterHand2.6M dataset <cit.> and the DeepHandMesh collection <cit.>. Experimental results show that XHand outperforms previous methods, providing high-fidelity meshes and real-time rendering of hands in various poses.





Our main contributions are summarized as follows:

     
  * A real-time expressive hand avatar with high fidelity results on both rendering and geometry, which is trained with an effective part-aware Laplace smoothing strategy.
     
  * An effective feature embedding module to simplify the training objectives and enhance the prediction accuracy by distinguishing invariant average features and pose-dependent features; 
     
  * An end-to-end framework to create photo-realistic and fine-grained hand avatars.  The promising results indicate that our method outperforms the previous approaches. 


The remainder of this paper is arranged as follows. Related works are introduced in Section <ref>. The proposed XHand
model and corresponding training process are thoroughly depicted
in Section <ref>. The experimental results and discussion are
presented in Section <ref>. Finally, Section <ref> sets out the conclusion of this paper
and discusses the limitations.


§ RELATE WORK







 §.§ Parametric Model-based Method

3D animatable human models <cit.> enable shape deformation and animation by decoding the low-dimensional parameters into a high-dimensional space. Loper et al. <cit.> introduce a linear model to explicitly represent the human body through adjusting shape and pose parameters. MANO hand model <cit.> utilizes a rigged hand mesh with fixed-topology that can be easily deformed according to the parameters. The low resolution of the template mesh hinders its application in scenarios requiring higher precision. To address this limitation, Li et al. <cit.> integrate muscle groups with shape registration, which results in an optimized mesh with finer appearance. Furthermore,
parametric model-based methods <cit.> have shown the promising results in accurately recovering hand poses from input images, however, they have difficulty in effectively capturing textures and geometric details for the resulting meshes. In this paper, our proposed XHand approach is able to capture the fine details of both appearance and geometry by taking advantages of Lambertian reflectance model <cit.>. 



 §.§ Model-free Approach

Parametric models have proven to be valuable in incorporating prior knowledge of pose and shape in hand geometry reconstruction <cit.>, while their representation capability is restricted due to the low resolution of the template mesh. To address this issue, Choi et al. <cit.> introduce a network based on graph convolutional neural networks (GCN) that directly estimates the 3D coordinates of human mesh from 2D human pose. Chen et al. <cit.> present a UV-based representation of 3D hand mesh to estimate hand vertex positions. Mobrecon <cit.> predicts hand mesh in real-time through a 2D encoder and a 3D decoder. Despite the encouraging results, the above methods still cannot capture the geometric details of hand. Moon et al. <cit.> propose an encoder-decoder framework that employs a template mesh to learn corrective parameters for pose and appearance. Although having achieved the improved geometry and articulated deformation, it has difficulty in rendering photo-realistic hand images. Gan et al. <cit.> introduce an optimized pipeline that utilizes multi-view images to reconstruct a static hand mesh. Unfortunately, it overlooks the variations due to joint movements. Karunratanakul et al. <cit.> design a shadow-aware differentiable rendering scheme that optimizes the abledo and normal map to represent hand avatar. However, its geometry remains smoothing. In contrast to the above methods, our proposed XHand approach is able to simultaneously synthesize the detailed geometry and photo-realistic images for drivable hands.





 §.§ Neural Hand Representation

There are various alternatives available for neural hand representations, such as HandAvatar <cit.>, HandNeRF <cit.>, LISA <cit.> and LiveHand <cit.>. In order to achieve high fidelity rendering of human hands, Chen et al. <cit.> propose HandAvatar to generate photo-realistic hand images with arbitrary poses, which take into account both occupancy and illumination fields. LISA <cit.> is a neural implicit model with hand textures, which focuses on signed distance functions (SDFs) and volumetric rendering. Mundra et al. <cit.> propose LiveHand that makes use of a low-resolution NeRF representation to describe dynamic hands and  a CNN-based super-resolution module to facilitate high-quality rendering. Despite the efficiency in rendering hand images, it is hard for those approaches to capture the details of hand mesh geometry. Luan et al. <cit.> introduce a frequency decomposition loss to estimate the personalized hand shape from a single image, which effectively address the challenge of data scarcity. Chen et al. introduce a spatially varying linear lighting model as a neural renderer to preserve personalized fidelity and sharp details under natural illumination. Zheng et al. facilitate the creation of detailed hand avatars from a single image by learning and utilizing data-driven hand priors. In this work, our presented XHand method focuses on synthesizing the hand avatars with fine-grained geometry in  real-time.



 §.§ Generic Animatable Objects

 
In addition to the aforementioned methods on hand modeling, there have been some studies reconstructing animatable whole or partial human bodies or animals <cit.>. Face models primarily pay their attention to facial expressions, appearance, and texture, rather than handling large-scale deformations <cit.>. Zheng at al. <cit.> bridge the gap between explicit mesh and implicit representations by a deformable point-based model that incorporates intrinsic albedo and normal shading. To build human body model <cit.>, numerous challenges arise from the intricate deformations, which make it arduous to precisely capture intricate details, such as textures and scan-like appearances, especially in smaller areas like the hands and face. Previous works <cit.> have explored to establish the correspondences between pose space and template space through linear blend skinning and inverse skinning weights. Alldieck et al. <cit.> employ learning-based implicit representations to model human bodies via SDFs. Chen et al. <cit.> propose a forward skinning model that finds all canonical correspondences of deformed points. Shen et al. <cit.> introduce XAvatar to achieve high fidelity of rigged human bodies, which employ part-aware sampling and initialization strategies to learn neural shapes and deformation fields. 


§ METHOD





Given multi-view images {I_t,i|i=1,...,N,t=1,...,T} for T frames captured from N viewpoints with pose {θ_t|t = 1,...,T} and shape β of their corresponding parametric hand models like MANO <cit.>, our proposed approach aims to simultaneously recover the expressive personalized hand meshes with fine details and render photo-realistic image in real-time. Fig. <ref> shows an overview of our method. Given the hand pose parameters θ, the fine-grained posed mesh is obtained from feature embedding modules (Sec. <ref>), which are designed to obtain Linear Blending Skinning (LBS) weights, vertex displacements and albedo by combining the average features of the mesh with pose-driven feature mapping. With the refined mesh, the mesh-based neural renderer achieves real-time photo-realistic rendering with respect to the vertex albedo ρ, normals 𝒩, and latent code Q in feature embedding modules.





 §.§ Detailed Hand Representation










In this paper, the parametric hand model MANO <cit.> is employed to initialize the hand geometry, which effectively maps the pose parameter θ∈ℝ^J×3 with J per-bone parts and the shape parameter β∈ℝ^10 onto a template mesh ℳ̅ with vertices V. Such mapping Ω is based on linear blending skinning with the weights W ∈ℝ^|V|× J. Thus, the posed hand mesh ℳ can be obtained by 

    ℳ = Ω(ℳ̅, W, θ, β).





Geometry Refinement. After increasing the MANO mesh resolution for fine geometry using the subdivision method in <cit.>, a personalized vertex displacement field D is introduced to allow the extra deformation for each vertex in the template mesh. The refined posed hand mesh ℳ_fine can be computed as below

    ℳ_fine = Ω(ℳ̅' + D, W', θ, β).


The original MANO mesh <cit.>, consisting of 778 vertices and 1538 faces, has limited capacity to accurately represent fine-grained details <cit.>. To overcome this challenge by enhancing the mesh resolution to capture intricate features, we employ an uniform subdivision strategy on the MANO template mesh, as shown in Fig. <ref>. By adding new vertices at midpoint of each edge for three times, we obtain a refined mesh with 49,281 vertices and 98,432 faces. To associate skinning weights with these additional vertices, we compute the average weights assigned to the endpoints of the corresponding edges.

Let 𝒮 denote the subdivision function for MANO mesh. The high resolution template mesh ℳ̅' and LBS weights W' can be extracted as follows

    ℳ̅', W' = 𝒮(ℳ̅, W).
  

To enhance the fidelity of the hand geometry, the vertex displacements D and the LBS weights W' are pose-dependent for each individual. This enables an accurate representation of the deformation under different poses. To this end, we propose the feature embedding modules Ψ_D and Ψ_lbs to better capture the intricate details of hand mesh, LBS weights W' are derived from the LBS embedding Ψ_lbs. The displacement embedding Ψ_D generates the vertex displacements D. Given the hand pose parameters {θ_t|t=1,...,T} for T frames, the mesh features are predicted as follows 
    D_t = Ψ_D(θ_t), W'_t = Ψ_lbs(θ_t).

Thus, the refined mesh ℳ_fine at time t can be formulated as below

    ℳ_fine = Ω(ℳ̅' + D_t, W'_t, θ_t, β).






Feature Embedding Module. Generally, it is challenging to learn the distinctive hand features in different poses. To better separate between the deformation caused by changes in posture and the inherent characteristics of the hand, we present an efficient feature embedding module in this paper. It relies on the average features of hand mesh and computes offsets of features in different poses, as illustrated in Fig. <ref>. 

Given a personalized hand mesh ℳ and its pose θ_t at time t, our feature embedding module extracts mesh features f_ℳ as follows 
    f_ℳ = Ψ(θ_t | f̅_ℳ),
 
where f̅_ℳ denotes the average vertex features of hand mesh. 





To represent the mesh features of personalized hand generated with hand pose θ_t, we design the following embedding function

    Ψ(θ_t | f̅_ℳ) = f̅_ℳ + Φ(θ_t, Q) * 𝒦,
 where Q is vertex latent code to encode different vertices. Φ denotes a pose decoder that is combined with multi-layer perceptrons (MLPs). It projects the pose θ_t and latent code Q onto the implicit space. To align with the feature space, 𝒦 is the mapping matrix to convert the implicit space ℝ^m into feature space ℝ^n, which subjects to 
    ∑_j=1^n𝒦_ij = 1,   for  i = 1, 2, …, m.


The personalized mesh features f_ℳ can be derived by combining the average vertex features f̅_ℳ and the pose-dependent offsets. Consequently, the LBS weights W_t' can be derived with average LBS weights f̅_lbs, pose decoder Φ_lbs, latent code Q_lbs and mapping matrix 𝒦_lbs as follows

    W_t' = Ψ_lbs(θ_t | f̅_lbs) = f̅_lbs + Φ_lbs(θ_t, Q_lbs) * 𝒦_lbs.
 Similarly, the vertex displacements D_t can be obtained as follows

    D_t = Ψ_D(θ_t | f̅_D) = f̅_D + Φ_D(θ_t, Q_D) * 𝒦_D,
 where f̅_D denotes average displacements. Φ_D, Q_D and 𝒦_D are pose decoder, latent code and mapping matrix for Ψ_D, respectively. The depths of Φ_lbs within the LBS embedding module Ψ_lbs and Φ_ρ within the albedo embedding module Ψ_ρ are set to 5, with each layer consisting of 128 neurons. Additionally, the depth of Φ_D within the displacement embedding module Ψ_D is 8, where the number of neurons is 512.

Remark. The feature embedding modules allows for the interpretable acquisition of hand features f_ℳ corresponding to the pose θ_t. The average mesh features are stored in f̅_ℳ, while the features offsets are affected by the pose θ. More importantly, the training objectives are greatly simplified by taking into account of the average features constraints, which leads to the faster convergence and improved accuracy.





 §.§ Mesh Rendering



Inverse Rendering. In order to achieve rapid and differentiable rendering of detailed mesh ℳ_fine, an inverse renderer is employed to synthesize hand images. Assuming that the skin color follows the Lambertian reflectance model <cit.>, the rendered image B can be calculated from the Spherical Harmonics coefficients 𝐆, the vertex normal 𝒩, and the vertex albedo ρ using the following equation 
    B(π^i) = ρ· SH(𝐆, 𝒩),
 where π^i is camera parameter of the i-th viewpoint. SH(·) represents Spherical Harmonics (SH) function of the third order. 𝒩 is the vertex normal computed from the vertices of mesh ℳ_fine. Similar to Eq. <ref>, the pose-dependent albedo ρ_t can be obtained from feature embedding module Ψ_ρ with average vertex albedo f̅_ρ, pose decoder Φ_ρ, latent code Q_ρ and mapping matrix 𝒦_ρ as follows

    ρ_t = Ψ_ρ(θ_t) = f̅_ρ + Φ_ρ(θ_t, Q_ρ) * 𝒦_ρ.

By analyzing how the variations in brightness relate to the hand shape, inverse rendering with the Lambertian reflectance model can effectively disentangle geometry and appearance.


Mesh-based Neural Rendering.
The NeRF-based methods usually employ volumetric rendering along its corresponding camera ray 𝐝 to acquire pixel color <cit.>, which usually require a large amount of training time. Instead, we aim to minimize the sampling time and enhance the rendering quality by making use of a mesh-based neural rendering method that is able to take advantage of the consistent topology of our refined mesh. 

The mesh is explicitly represented by triangular facets so that the intersection points between rays and meshes are located within the facets. The features that describe meshes, such as position, color, and normal, are associated with their respective vertices. Consequently, the attributes of intersection points can be calculated by interpolating the three vertices of triangular facet to its intersection point. The efficient differentiable rasterization <cit.> ensures the feasibility of inverse rendering and mesh-based neural rendering.





Given a camera view π^i, our mesh-based neural render 𝒞(π^i) synthesizes the image with respect to the position 𝐱, normal 𝒩, feature vector 𝐡 and ray direction 𝐝, where 𝐱, 𝐡 and 𝒩 are obtained through interpolating with ℳ_fine. 𝐡 in neural render 𝒞(π^i)  contains the latent codes Q_D and Q_ρ detached from Ψ_D and Ψ_ρ, and feature vector Q_render <cit.>. Q_render is utilized to represent the latent code of vertices during rendering. As in <cit.>, the neural network 𝒞 comprises 8 fully-connected layers with ReLU activations and 256 channels per layer, excluding the output layer. Furthermore, it includes a skip connection that concatenates the input to the fifth layer, which is depicted in Fig. <ref>.







 §.§ Training Process



To obtain a personalized hand representation, the parameters of the three feature embedding modules Ψ_D, Ψ_lbs, and Ψ_ρ, as well as the neural render 𝒞, require to be optimized based on multi-view image sequences. Our training process consists of three steps, including initialization, training feature embedding modules, and training the mesh-based neural render.

Initialization of XHand.
To train our proposed XHand model, the average features f̅_ℳ of mesh in feature embedding significantly affect training efficiency and results. Random initialization has great impact on training due to estimation errors in Ψ_lbs and Ψ_D, which m ay lead to the failure of inverse rendering. Therefore, it is crucial to initialize the neural hand representation. To this end, the reconstruction result of the first frame (t=1) is treated as the initial model. 

Inspired by <cit.>, XHand model is initialized from multi-view images. The vertex displacement D and vertex albedo ρ of hand mesh are jointly optimized through inverse rendering. Mesh generation is obtained from Eq. <ref>, and the rendering equation is same as Eq. <ref>. The loss function during initialization is formulated as below 
    ℒ_init = ∑_i||B(π^i) - I_i||_1 + ∑ L × D + ∑ L ×ρ,
 where L is the Laplacian matrix <cit.>. Laplacian terms L × D and L ×ρ are employed to regularize the mesh optimization, as the mesh features are supposed to be smooth. Uniform weights of the Laplacian matrix are adopted in training. The outcomes D and ρ are used to initialize Ψ_D and Ψ_ρ. The initialization of Ψ_lbs is directly derived from MANO model <cit.>.

Loss Functions of Feature Embedding.
Inverse rendering is utilized to learn the parameters of three feature embedding modules Ψ_D, Ψ_lbs and Ψ_ρ. ℒ_inv is introduced to minimize the errors of rendering images as follows
    ℒ_inv = ℒ_rgb + ℒ_reg,
 where ℒ_rgb represents the rendering loss. ℒ_reg is the regularization term. Inspired by <cit.>, we use L_1 error combined with an SSIM term to form the ℒ_rgb as below 
    ℒ_rgb   = λ∑_i||B(π^i) - I_i||_1 + (1-λ)ℒ_SSIM(B(π^i), I_i),
 where λ denotes the trade-off coefficient. 

To enhance the efficiency in extracting geometric information from images, we introduce the part-aware Laplace smoothing term ℒ_pLap. The Laplace matrix 𝐀 of mesh feature f is defined as 𝐀 = L × f. Hierarchical weights ϕ_pLap are introduced to balance the weights of regularisation via different levels of smoothness. φ_i in matrix ϕ_pLap is defined as follows 
    φ_i = {[   γ_1   0 < 𝐀_i < p_1; γ_2   p_1 < 𝐀_i < p_2;                   ... ].
    ,
 where {p_1, p_2,…} represent the threshold values for the hierarchical weighting and {γ_1, γ_2,…} denote the balanced coefficients. The part-aware Laplace smoothing ℒ_pLap is used to reduce excessive roughness in albedo and displacement without affecting the fine details, which is defined as follows 
    ℒ_pLap(f) = ∑_iϕ_pLap𝐀.
 By employing varying degrees of hierarchical weights to trade-off Laplacian smoothing, ℒ_pLap is able to better constrain feature optimization in different scenarios. In our cases, minor irregularities are considered to be acceptable, while excessive changes are undesirable. Therefore, the threshold p can be dynamically controlled through the quantiles of Laplace matrix A, where those greater than p will be assigned larger balance coefficients.

The following regularization terms are introduced to conform the optimized mesh to the hand geometry

    ℒ_reg  = ℒ_pLap(ρ) + ℒ_pLap(D)   + α_1ℒ_mask + α_2ℒ_e + α_3ℒ_d.
 
where ℒ_pLap(ρ) and ℒ_pLap(D) are part-aware Laplacian smoothing terms to maintain albedo and displacement flattening during training. ℒ_mask, ℒ_e and ℒ_d are utilized to ensure that the optimized hand mesh remains close to the MANO model, where each term is assigned with constant coefficients denoted by α_1, α_2 and α_3. Let ℒ_mask = ∑_i||M̂ - M||_1 represents the L_1 loss between the mask M̂ rendered during inverse rendering and the original MANO mask. ℒ_e penalizes the edge length changes of e_ij  with respect to MANO mesh as ∑_i,j||ê_ij - e_ij||_2^2, where ê_ij is the Euclidean distance ||·||_2^2 between adjacent vertices V_i and V_j on the mesh edges. e_ij denotes the edge distance of the subdivided MANO mesh ℳ̅'. 
ℒ_d = ∑_i||D_i||_2^2 is employed to constrain the degree of displacement.

Loss Functions of Neural Renderer.
Once the latent codes Q_D and Q_ρ of Ψ_D and Ψ_ρ are detached, ℒ_neu is used to minimize the residuals between the rendered image and the ground truth like Eq. <ref>
    ℒ_neu = ω∑_i||𝒞(π^i) - I_i||_1 + (1-ω)ℒ_SSIM(𝒞(π^i), I_i),

where ω denotes balanced coefficient.


§ EXPERIMENTS








 §.§ Datasets


InterHand2.6M. The InterHand2.6M dataset <cit.> is a large collection of images, each with a resolution of 512×334 pixels, accompanied by MANO annotations. It includes multi-view temporal sequences of both single and interacting hands. The experiments primarily utilize the 5 FPS version of this dataset.

DeepHandMesh. The DeepHandMesh dataset <cit.> features images captured from five different viewpoints, matching the resolution of those in InterHand2.6M. It also provides corresponding 3D hand scans, facilitating the validation of mesh reconstruction quality against 3D ground truth data.



 §.§ Experimental Setup


Implementation Details. In the experiments, our proposed XHand model is mainly trained and evaluated on the 5FPS version of Interhand2.6M dataset <cit.>, which is made of large-scale multi-view sequences capturing a wide range of hand poses. Each sequence has dozens of images with the size of 512×334. As in <cit.>, XHand model is trained on the InterHand2.6M dataset with 20 views across 50 frames for each sequence. The remaining frames are used for evaluation. To assess the quality of mesh reconstruction, we conduct experiments on DeepHandMesh dataset <cit.>, which consists of 3D hand scans along with images captured from five different views. The images are with the same size as those in InterHand2.6M dataset. We conducted all the experiments on a PC with NVIDIA RTX 3090 GPU having 24GB GPU memory.




We employ PyTorch and Adam Optimizer with a learning rate of 5e^-4. To facilitate differentiable rasterization, we make use of the off-the-shelf renderer nvdiffrast <cit.>. As in <cit.>, positional encoding is performed on 𝐝 and 𝐱 before feeding them into the rendering network. In our training process, the feature embedding modules are firstly trained for 500 epochs using inverse rendering. Then, feature embedding modules and neural render are jointly trained for 500 epochs, where the average features f̅_ℳ in feature embedding modules are updated every 50 epochs. We empirically found that the best performance is achieved in case of λ=ω=0.8, α_1=10, α_2=1e^5, and α_3=1e^4. To avoid the excessive displacements and color variations, in ℒ_pLap(ρ), p_1 is set to the first quartile of A_ρ, γ_1 is set to 0.1, and γ_2 is 1. Similarly, in ℒ_pLap(D), p_1 is the median of A_D, and γ_1 = 0.1, γ_2 = 20. The lengths of latent codes Q_lbs, Q_D, Q_ρ and Q_render are set to 10, 10, 10 and 20, respectively.

Evaluation Metrics. In the experiments, we fit the hand mesh representations to multi-view images sequence for single scene. For fair comparison, we employ the same evaluation metrics as in <cit.>, which measure the synthesized results with peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), and learned perceptual image patch similarity (LPIPS). We calculate the average point-to-surface Euclidean distance (P2S) to assess the accuracy of the reconstructed hand mesh, which is measured in millimeters since the Chamfer distance metric is considered unsuitable due to scale variations between MANO and 3D scans.



 §.§ Experimental Results




To investigate the efficacy of our proposed XHand, we treat the subdivided MANO model <cit.> with vertex albedo as our baseline, which has the merits of the efficient explicit representation. Moreover, we compare our model against several rigged hand expression methods, including LISA <cit.>, HandAvatar <cit.>, HandNeRF <cit.>, and LiveHand <cit.>. For fair comparison, LiveHand is re-trained with the same setting and LISA is reproduced by <cit.>.





We firstly perform the quantitative evaluation on rendering quality, as shown in Table <ref>. The evaluation metrics of LISA <cit.> are adopted from LiveHand <cit.> and the results of HandNeRF <cit.> are obtained from their original paper. It can be seen that our proposed XHand approach achieves the best results with a PSNR of 34.3dB. Our baseline drives a textured MANO model through LBS weights. Due to lacking the ability to handle illumination changes across different scenes and poses, there exist some artifacts with a PSNR of 28.6dB. NeRF-based methods <cit.> present the competitive PSNR results, which rely on MANO mesh without fine-grained geometry during rendering. By taking advantage of fine-grained meshes estimated by XHand, our method outperforms the previous approaches using volumetric representation in terms of the rendering quality. Benefiting from our design, XHand achieves 56 frames per second (FPS) on inference. Specifically, the feature embedding modules require 0.7 milliseconds, inverse rendering requires 15 milliseconds and the neural rendering module needs 0.1 milliseconds.






Table <ref> shows the results on DeepHandMesh dataset. Our method outperforms the annotated MANO mesh <cit.> and DHM <cit.> by 3.3 mm and 1.2 mm on P2S. This indicates that our proposed feature embedding module can accurately capture the underlying hand mesh deformation comparing to the encoder-decoder scheme in DHM.
More experimental results conduct on the DeepHandMesh <cit.> dataset are visualized in Fig. <ref>. 




For better illustration, Fig. <ref> shows the more detailed comparisons of rendering and geometry on InterHand2.6M test split. Due to the limited expressive capability, it is hard for the baseline MANO model <cit.> to capture muscle details varying across different poses. Although the hand meshes generated by HandAvatar <cit.> have more details than MANO, they are still smoothing compared to ours. In terms of geometry, our method exhibits more prominent skin wrinkles based on different poses. The NeRF-based method HandNeRF <cit.> and LiveHand <cit.> yield the competitive render results, while they still rely on the MANO model and cannot obtain fine-grained hand geometry. On the contrary, our approach effectively presents an accurate hand representation by taking advantage of the feature embedding module and the topological consistent mesh model, resulting in enhanced rendering quality and geometry quality. Fig. <ref> visualizes the results of different identities animated using reference poses. 

 

The proposed method efficiently drives personalized hand expressions from arbitrary hand gesture inputs. To demonstrate its performance, in-the-wild data serve as a reference for hand poses, as illustrated in Fig. <ref>. The pose parameters of in-the-wild videos are extracted from HaMeR <cit.>.
It is worth noting that we can enhance the vividness of the images by using different spherical harmonic coefficients for relighting.  

























 §.§ Ablation Study





We perform extensive ablation experiments on the InterHand2.6M dataset test set to validate the contributions of various modules and settings within our framework. First, we aim to demonstrate the performance improvements achieved by our proposed feature embedding module and part-aware Laplace smoothing strategy, consistent with our design intentions for the fusion modules. Second, we intend to showcase the robust performance of our XHand model across different numbers of views, highlighting its effectiveness even with limited viewpoints. Furthermore, we conduct a comparative analysis of various neural rendering networks. Based on this evaluation, we have chosen MLPs to enhance both the inference speed and the rendering quality, ensuring efficient and high-fidelity output. The following sections detail these ablation experiments and analyze the results comprehensively.

Ablation Study on Different Components. In the first row of Fig. <ref>, it can be seen that our method significantly highlights skeletal movements and skin changes. Moreover, our design resolves the issue of lighting variations. Our proposed part-aware Laplacian regularization effectively reduces the surface artifacts without sacrificing the details. The feature embedding modules are able to guide the learning of hand avatars by distinguishing average features and pose features, which enhance the reconstruction accuracy.









































 

Table <ref> shows that the level of mesh detail significantly affects image quality. The rendering results are substantially enhanced through feature embedding. The part-aware Laplacian regularization yields more realistic geometric results, indirectly improving the accuracy of the neural render. Furthermore, the Position Encoder in neural rendering leads to better image quality. 

Ablation Study on Number of Views. Typically, the performance of each model is improved along with the increasing number of input images, particularly for the NeRF-based methods. Also, insufficient training data may lead to the reconstruction failure. We conducted ablation experiments using different numbers of views as inputs. As shown in Table <ref>, we trained the model on sequences of 1, 5, 10, 20 and 30 views to demonstrate the impact of views. Despite being trained with a limited number of viewpoints, including as few as a single viewpoint, our method effectively captures the hand articulations. Furthermore, we achieve the competitive results in case of more than 10 input views.




Choices of Neural Rendering. Traditional neural radiance fields <cit.> typically employ 8-layer MLPs as the renderer. In contrast, our mesh-based network eliminates the necessity for point cloud sampling, which is able to render through vertex features. Benefiting from topology consistency, our neural renderer can make use of UNet <cit.> which leads to promising performance. To explore this, we conduct ablation experiments on both network architectures, as detailed in Table <ref>. These experimental results demonstrate that a UNet with 4 layers achieves superior rendering quality, albeit at the expense of inference speed. In comparison to UNet, MLPs can enhance performance by 20% with only a marginal loss in accuracy. Therefore, we have chosen to employ MLPs as our neural renderer. Furthermore, our investigation into a well-designed image generation network, EG3D <cit.>, reveals its unsuitability for neural rendering. 







§ CONCLUSION


We present XHand, a real-time expressive hand avatar with photo-realistic rendering and fine-grained geometry. By taking advantage of the effective feature embedding modules to distinguish average features and pose-dependent features, we obtain the finely detailed meshes with respect to hand poses. To ensure the high quality of hand synthesis, our method employs a mesh-based neural render that takes consideration of mesh topological consistency. During the training process, we introduce the part-aware Laplace regularization to reduce the artifacts while maintaining the details through different levels of regularization. Rigorous evaluations conducted on the InterHand2.6M and DeepHandMesh datasets demonstrate the ability to produce high-fidelity geometry and texture for hand animations across a wide range of poses.




Our method relies on accurate MANO annotations provided by the dataset during training. For future work, we will consider to explore the effective MANO model parameter estimator. 







IEEEtran



[
    < g r a p h i c s >
]Qijun Gan is currently a PhD candidate in the College of Computer Science and Technology, Zhejiang University, Hangzhou, China. Before that, he received the bachelor degree from University of International Business and Economics, China. His research interests include machine learning and computer vision, with a focus on 3D reconstruction.

[
    < g r a p h i c s >
]Zijie Zhou received the B.S. degree in Communication Engineering from Beijing University of Post and Telecommunication, Beijing, China, in 2022. He is currently a postgraduate student in the School of Software Technology, Zhejiang University, Hangzhou, China. His research interests include computer vision and deep learning.


[
    < g r a p h i c s >
]Jianke Zhu  received the master’s degree from University of Macau in Electrical and Electronics Engineering, and the PhD degree in computer science and engineering from The Chinese University of Hong Kong, Hong Kong in 2008. He held a post-doctoral position at the BIWI Computer Vision Laboratory, ETH Zurich, Switzerland. He is currently a Professor with the College of Computer Science, Zhejiang University, Hangzhou, China. His research interests include computer vision and robotics. 





