Paper Title: PhysFlow: Skin tone transfer for remote heart rate estimation through conditional normalizing flows.



The Impacts of AI Avatar Appearance

and Disclosure on User Motivation
    Boele Visser10009-0002-7504-6495 Peter van der Putten10000-0002-6507-6896 Amirhossein Zohrehvand10000-0002-7473-6702
    August 5, 2024
========================================================================================================================




§ ABSTRACT 








In recent years, deep learning methods have shown impressive results for camera-based remote physiological signal estimation, clearly surpassing traditional methods. However, the performance and generalization ability of Deep Neural Networks heavily depends on rich training data truly representing different factors of variation encountered in real applications. Unfortunately, many current remote photoplethysmography (rPPG) datasets lack diversity, particularly in darker skin tones, leading to biased performance of existing rPPG approaches. To mitigate this bias, we introduce PhysFlow, a novel method for augmenting skin tone diversity in remote heart rate estimation using conditional normalizing flows. PhysFlow adopts end-to-end training optimization, enabling simultaneous training of supervised rPPG approaches on both original and generated data. Additionally, we condition our model using CIELAB color space skin features directly extracted from the facial videos without the need for skin-tone labels. We validate PhysFlow on publicly available datasets, UCLA-rPPG and MMPD, demonstrating reduced heart rate error, particularly in dark skin tones. Furthermore, we demonstrate its versatility and adaptability across different data-driven rPPG methods.





§ INTRODUCTION







In recent years, the interest on camera-based measurement of physiological signals has undergone substantial growth caused by its potential applicability in clinical <cit.> and human-computer interaction applications <cit.>. The recent advancements in deep learning have further propelled this field. However, data-driven methods often require a large amount of training data to achieve good generalization, making their performance 
highly dependent on the distribution and scale of the dataset chosen for training. Additionally, many rPPG datasets suffer from biases, particularly regarding demographic diversity, leading to unfair performance depending on skin tone, especially for underrepresented ethnicities. 

Previous studies <cit.> have investigated the effects of demographic biases in existing rPPG datasets, revealing a significant decline in remote heart rate estimation performance for subjects with the darkest skin tones. While these studies highlighted the consequences of skin tone bias, none of them proposed concrete solutions to address the problem. Although the creation of datasets that accurately represent all skin types may seem like a simple solution to overcome this problem, it is often impractical and resource-intensive, especially during the recruiting of the participants. Consequently, each dataset typically over-represents the population of the country in which it was collected. For example, datasets like VIPL-HR <cit.> or PURE <cit.> predominantly consist of Asian or Caucasian populations, respectively.

Only a few recent studies have addressed skin tone bias. Two of them <cit.> used synthetic avatars to tackle data imbalance. However, these approaches are often not photo-realistic and can be time-consuming in data generation. Another work <cit.> employed radar hardware to improve rPPG signal recovery for dark skin tones. Nevertheless, this method is not always feasible and limits rPPG versatility with conventional cameras. Finally, Ba et al. <cit.> proposed a two-stage optimization using external pre-training <cit.> to transfer skin tone content, which is an interesting possibility to balance skin-tone distributions as long as external data with the underrepresented skin tones are available for pre-training. 


In this context, we propose a novel data augmentation approach leveraging normalizing flows <cit.> to disentangle the skin tone information from the rest of the facial video content. Additionally, 
they allow for attribute control by concatenating parameters to embeddings, facilitating the conditioning of specific properties such as skin tone. Furthermore, in this paper, we approach the representation of skin tone for rPPG estimation differently than before. Previous works typically used the Fitzpatrick scale <cit.> to evaluate or annotate skin tone, categorizing it into six levels from I (lightest) to VI (darkest). In contrast, we propose using a bi-dimensional representation <cit.> to quantify the apparent skin color, which extracts the perceptual light and hue angle from facial videos, offering three key advantages over the Fitzpatrick scale. Firstly, it enables automated skin tone measurement in each facial video, eliminating the need for manual annotations and allowing to use it into unlabeled data, common in the majority of rPPG datasets. Secondly, it simplifies the collection and annotation for new rPPG datasets, reducing human error inherent in manual Fitzpatrick scale annotations due to subjective perceptions. Lastly, unlike the Fitzpatrick scale, which considers only lightness, this bi-dimensional representation accounts for variations in hue, distinguishing between red and yellow hues. This broader perspective better accommodates diverse skin tones, particularly given that common experiments in existing rPPG datasets involve varying lighting conditions using different types of light sources.

The main contributions of the paper are three-fold: 

    
  * We introduce PhysFlow, a novel skin tone data augmentation approach for rPPG estimation, which adopts conditional normalizing flows to disentangle skin tone information from other appearance and temporal facial video features.
    
  * We propose a novel training that leverages skin tone CIELAB color space representation without requiring external labels.
    
  * We train the PhysFlow model end-to-end to optimize any supervised rPPG approach using the original and generated data within the same data generation process promoting a fast adaptation.
    





§ RELATED WORK


Camera-based PPG measurement. Since Takano et al. <cit.> and Verkruysse et al. <cit.> demonstrated remote HR measurement feasibility from facial videos, diverse methods for physiological data recovery have emerged. Some focus on regions of interest, employing techniques like Blind Source Separation <cit.>, Normalized Least Mean Squares <cit.>, or self-adaptive matrix completion <cit.>. Others utilize the skin optical reflection model, projecting RGB skin pixel channels into an optimized subspace to mitigate motion artifacts <cit.>. Deep learning-based methods <cit.> have surpassed conventional techniques, achieving state-of-the-art performance in estimating vital signs from facial videos. Some combine traditional methods with Convolutional Neural Networks (CNNs) to leverage advanced features <cit.>. Other recent works <cit.> explore unsupervised approaches using meta-learning, enhancing generalization in out-of-distribution cases. Unlike previous approaches, some propose end-to-end models <cit.> to directly extract the rPPG signal from facial videos. Transformer-based models like Physformer <cit.> and RADIANT <cit.> have gained attention for leveraging long-range spatiotemporal features, although they currently lack optimization for mobile deployment. While promising, they may not yet demonstrate a significant performance advantage over CNN-based models <cit.>. Finally, works like <cit.> and <cit.> propose lightweight rPPG frameworks with competitive HR results while controlling computational cost.

Skin tone bias in rPPG measurement. Earlier works <cit.> noted the lowest signal-to-noise ratio and highest error rates for darker skin tones in assessments on private datasets. Prior to the work of Nowara et al. <cit.>, the influence of skin tone on rPPG extraction had only been examined in limited datasets without population diversity and data-driven approaches. In their meta-analysis, Nowara et al. investigated the impact of skin type on rPPG signal recovery for each Fitzpatrick skin tone using three datasets. They found significantly poorer rPPG signal recovery for subjects with category VI compared to other skin types. In <cit.>, a new color space for rPPG estimation was proposed, showing consistent results across all skin tones, albeit with a performance drop for darker skin tones (V and VI), similar to <cit.>. Subsequently, Dasari et al. <cit.> conducted an extensive study analyzing estimation biases of camera PPG methods across diverse demographics. They observed similar biases as with contact-based devices and environmental conditions. 

rPPG data-augmented approaches. While various works have identified skin tone bias in rPPG estimation, only a few solutions have directly addressed its mitigation. Recent studies have proposed different data augmentation approaches considering factors like motion <cit.> or data scarcity <cit.>, but without targeting skin tone bias. Two recent works aimed to balance demographic groups present in current rPPG datasets. Firstly, Vilesov et al. <cit.> proposed a method to mitigate bias across skin tones using multi-modal fusion, combining information from an RGB camera and 77 GHz radar. However, the need for a radar device limits the applicability of camera-based measurements in all scenarios. Alternatively, Ba et al. <cit.> introduced a two-step joint optimization framework to translate light-skinned to dark-skinned facial videos while maintaining their pulsatile signals, resulting in a considerable reduction of HR estimation error in their private dataset. However, this approach has two main limitations. Firstly, the use of a predefined model <cit.> to translate skin tone is limited to only four labels (African, Asian, Caucasian, and Indian) and may not discern different variations for each skin label. Secondly, the two-stage joint optimization can lead to unrealistic skin tone transformations, with certain facial regions appearing more illuminated than others. In contrast, our novel approach is independent of external pre-training and can translate skin tone from the same data or a reduced set of data. Our end-to-end approach using conditional normalizing flows produces more homogeneous, realistic, and diverse skin tone transfers. Also some studies have tackled the limited availability of data in existing benchmarks by generating synthetic data. For example, McDuff et al. <cit.> used Blender to create synthetic avatars aligned with cardiac and respiratory signals across various scenarios, while Wang et al. <cit.> employed a 3D morphable face model to generate synthetic facial videos. Although both methods show promise, they have limitations. Primarily, the generation of synthetic data can be excessively time-consuming, especially in the case of <cit.>, and they may suffer from domain shift, where models trained on synthetic data fail to perform well on real datasets due to the domain gap between the datasets, requiring a domain adaptation stage <cit.>. Additionally, in both instances, the facial appearance lacks photo-realism, and certain physiological factors like Pulse Transit Time (PTT) are not considered. In contrast, our data augmentation approach proposes direct training that simultaneously trains the rPPG model while generating new data, preserving the photo-realism of the original data and the characteristics of pulsatile data.



§ METHODOLOGY





 §.§ Preliminaries: Continous Normalizing Flows


Normalizing flows (NFs) <cit.> are a class of generative neural networks that estimate an unknown complex data distribution, p(z), from a known and simpler base probability distribution, p(u), such as a normal distribution 𝒩(μ,σ^2). The mapping between both distributions is achieved by applying a sequence of bijective and differentiable transformations f_1, ...,f_K : ℝ^d→ℝ^d. Additionally, the mapping between both distributions can be constrained to some set of conditions or attributes c, these types of NFs are commonly known as conditional normalizing flows (c-NFs). Hence, the relationship between z|c and u|c can be defined as:

    u= f_θ(z;c),   z= f^-1_θ(u;c)


where u∼ p(u|c), z∼ p(z|c), and f_θ is a bijective neural network parameterized by θ.  Mathematically, c-NFs can express the log-likelihood by applying the change of variables rule:  

    log p(z; c, θ) = log p(f_θ(z;c)) - ∑_i=1^k-1log | det(J_θ_i(z; c)|

where  J_θ_i(z; c)=∂ f_θ_i(z;c)/∂ z is the Jacobian, and |det(J_θ_i(z; c)| represents the volume change caused by the transformations of f_θ. 

The normalizing flow can be generalized from a discrete sequence to a continuous transformation <cit.>, parameterizing the dynamics of data transformation over time ∂ u_τ/∂τ = g_θ(τ,c,u_τ), where g_θ is a neural network, u_τ is the state at time τ, u ∼𝒩(μ,σ^2) and c is the conditioning attributes. The Continuous Normalizing Flow (CNF) computes z from u by integrating g_θ over time, express as:


    z = u_τ_0 + ∫_τ_0^τ_1 g_θ(τ,c,u_τ) dτ


Following the Instantaneous Change of Variables theorem <cit.> where the change in log probability of a continuous random variable is equal to the trace of the Jacobin matrix, the total change in log density can be written as: 

    log p(u(τ_1)) = log p(u(τ_0)) - ∫_τ_0^τ_1 Tr(∂ g_θ/∂ u_τ) dτ


To compute the gradients with respect to the parameters a black-box ordinary differential equation (ODE) solver is applied using an adjoint sensitivity method <cit.>. In this work, we adopt CNFs because they are demonstrated to have better expressiveness and versatility than discrete NFs <cit.>. 




 §.§ PhysFlow



The proposed PhysFlow approach, outlined in Fig.<ref>, aims to mitigate the effect of skin tone bias in rPPG data-driven models by generating a diverse range of skin tone-augmented facial videos through the usage of CIELAB skin tone features. Our model comprises three main components: a 3D auto-encoder (AE), a conditional CNF network, and the rPPG model.




  §.§.§ 3D video reconstruction


Inspired by several flow-based generative models <cit.> in different computer vision tasks, we employ a conventional auto-encoder model, which is trained to encode and decode the high-dimensional information into low-dimensional feature embedding, independently of any disentanglement. Unlike previous works, we aim to preserve the underlying physiological signals that depend on subtle spatio-temporal variations. Therefore, the preservation of inter-frame content is crucial to recover the rPPG signal, thus we choose a 3D-CNN AE to model the spatio-temporal information of the input facial videos (the architecture details are explained in supplementary material). During the 3D-CNN AE training, the features are only spatially downsampled and upsampled in order to not degrade the temporal content. To train the proposed 3D-CNN AE, we utilize the following combined loss:  


    ℒ_AE = α_1ℒ_rec + α_2ℒ_perc +α_3ℒ_rPPG,

    
    ℒ_rec = X̂_t- X_t _1, 
            ℒ_perc =  1/C_j H_j W_jϕ_j(X̂_t))-ϕ_j(X_t)) _1, 
            ℒ_rPPG =   rPPG_rec-rPPG_ori_2


where α_*, controls the importance between different loss
components and ℒ_rec is the reconstructed loss between the reconstructed X̂_t and original X_t facial videos computed by the L_1-loss. The ℒ_perc is a VGG perceptual loss <cit.> adopted to retain high-frequency details between the reconstructed and original video, where ϕ_j represents the perceptual function which outputs the activation function of the jth layer in the VGG network and C_j, H_j, W_j are the dimensions of the tensor feature map. Finally, ℒ_rPPG is a L_2-loss between the extracted rPPG in the original and reconstructed facial video to ensure the preservation of the physiological data.



  §.§.§ Skin tone representation and conditioning


Our data augmentation approach relies on two primary assumptions. Firstly, for effective augmentation, the unbalanced dataset should feature a minimum level of skin tone diversity, enabling the transfer of skin tone without relying on an external dataset. Secondly, consistent luminance conditions within the same video are necessary for successful skin tone transfer. Hence, in this work, we select the UCLA-rPPG dataset <cit.>. Despite its significant imbalance in skin tone representation, the dataset offers sufficient diversity and samples, while maintaining constant illumination across each facial video. 
Based on these assumptions, we model our skin tone transfer employing a bi-dimensional representation <cit.> expressed in CIELAB color space through two components: luminance and hue. The luminance component determines the lightness or darkness of the skin, while the hue component, derived by dividing the alpha and beta channels, spans from red to yellow. Fig. <ref> illustrates the comparison between our bi-dimensional skin tone label representation and the annotated Fitzpatrick labels from the UCLA-rPPG dataset. We notice that the luminance and hue values are not well captured by the subjective Fitzpatrick annotations, i.e. some subjects annotated in a specific skin type may belong to another one, particularly between skin types I to V, as seen in Fig. <ref>. 
These subjective annotations may stem from the challenge of labelling a diverse range of skin tones within only six main classes, where considerable diversity exists within each class. Therefore, the use of CIELAB skin tone labels not only facilitates the skin tone transfer without relying on manual Fitzpatrick labels but also ensures the ability to map large variations of skin tones and overcome illumination changes between different trials on the same subject, which could affect the perception of skin type.

To apply skin tone transfer, once the 3D auto-encoder has converged, we freeze its parameters and proceed to train the whole system. During the end-to-end training, the c-CNF module is responsible for mapping the entangled latent representation to a disentangled form. The invertibility of the flow-based model ensures that no information is lost during the transition from the entangled to the disentangled latent space. To achieve skin tone disentanglement, our CNF model is conditioned on luminance and hue values computed for a sequence of frames. 


    < g r a p h i c s >





  §.§.§ Model training and optimization


For training the PhysFlow model, we utilize end-to-end training, optimizing both the c-CNF module and the rPPG approach simultaneously. This involves minimizing the continuous form of the negative log-likelihood, as previously introduced in Equation <ref>:


    ℒ_NLL =  - log p(u(τ_0)) + ∫_τ_0^τ_1 Tr(∂ g_θ/∂ u_τ) dτ


During the PhysFlow training, we also control the synthesis of the skin tone transfer by conditioning the c-CNF with specific transferable skin tone values and assessing the generated output. For that, we introduce three regularization losses. Firstly, we compute a consistency color loss ℒ_color between the conditioned skin tone values st_cond and the resulting skin tone measured from the generated video ŝt̂_pred. Secondly, to enforce the preservation of the temporal content we consider a temporal consistency loss ℒ_temp, which is computed by minimizing the error of the first-order derivative of the original z_video and the conditioned ẑ_video latent space:   


    ℒ_color = ŝt̂_pred - st_cond_1, 
            ℒ_temp =  ∂ẑ_video/∂ t - ∂ z_video/∂ t_1


Lastly, we use the same perceptual loss ℒ_perc from Eq. <ref> to keep the overall appearance of the original face. 
At the same time, to optimize the supervised rPPG approach we incorporate a physiological loss which combines the minimization of the rPPG signal concerning the ground truth PPG from the original and also from the generated video, which can be written as:


    ℒ_phys = ℒ_PPGgen +ℒ_PPGori

To compute ℒ_phys, in our experiments, we use the Negative Pearson correlation <cit.> and TALOS <cit.> losses depending on the selected rPPG approach. Finally, we can express the overall training objective for PhysFlow as the minimization of the sum of all the losses: 


    ℒ_overall = λ_1ℒ_NLL + λ_2ℒ_perc +λ_3ℒ_color +λ_4ℒ_temp +λ_5ℒ_phys


where λ_*, controls the importance between different loss components. 






§ EXPERIMENTS





 §.§ Experimental Setup

Datasets and models: We consider two recent datasets, UCLA-rPPG <cit.> and MMPD <cit.>. Both benchmarks consist of facial videos and corresponding gold-standard PPG signals, while also containing Fitzpatrick labels  (see supplementary materials for more details). To evaluate PhysFlow performance, we use three deep-learning models: PhysNet <cit.>, EfficientPhys <cit.>, and DPMN+TDM <cit.>. For our experiments, we use the UCLA-rPPG dataset for training and MMPD for cross-dataset evaluation. Also, we include some traditional methods such as POS <cit.> and CHROM <cit.> as baseline results.



Metrics and evaluation protocol: To evaluate the HR estimation performance of the proposed model, we adopt the same metrics used in the literature, such as the mean absolute HR error (MAE), the root mean squared HR error (RMSE), the mean absolute percentage error (MAPE) and Pearson’s correlation coefficients R <cit.>. Our experiments are performed using subject evaluation with a 10-sec sequences average with no overlap to compute HR estimation for all reported metrics. For our cross-dataset experiment, we utilize the MMPD dataset considering only the skin tone. For this reason, we followed the same protocol in <cit.> to exclude from our analysis other factors such as exercise or lighting conditions. 

Implementation details:  In all our experiments, we utilize the Mediapipe Face Mesh model <cit.> to focus our analysis only on facial skin pixels. After masking the facial video, each frame is resized to 96× 96 pixels. The PPG ground truth is pre-processed following <cit.> to denoise the raw PPG signal, which facilitates a better model convergence during the training. We use PyTorch 2.1.2 <cit.> and train on a single NVIDIA RTX3060 using sequences of 300 frames without overlap and AdamW optimizer with a learning rate of 0.0001. As explained in section <ref>,  we first pre-trained the 3D AE for 310 epochs and then froze the weights. Following <cit.>, during the 3D AE training, we add a small Gaussian noise of 0.05 to the latent space to prevent the latent distribution from collapsing to single delta peaks. Furthermore, for fast convergence, once the 3D AE is trained we also pre-trained the c-CNF module for 50 epochs and then trained end-to-end jointly with the supervised rPPG model. The weights for different losses in 3D AE training are set as α=500, α=10 and α=5, while in PhysFlow training are set as λ_1=1, λ_2=100, λ_3=1000, λ_4=0.0001 and λ_5=10.
Finally, the predicted rPPG signal is filtered using a band-pass filter with cut-offs of 0.66-3Hz while the heart rate is calculated using Chirp-Z Transform (CZT) <cit.>.



 §.§ Experimental Results


Table <ref> summarizes the assessment of remote HR performance across different Fitzpatrick scale splits in MMPD. These baseline evaluations are conducted using both traditional and data-driven approaches. From these results, several conclusions can be drawn. Firstly, across all methods, there is a noticeable decline in heart rate performance with darker skin tones, consistent with findings in previous studies <cit.>. Despite the similar representation of skin types IV, V, and VI between the datasets, a significant performance gap is evident, potentially influenced by their proximity to the more prevalent types such as skin tones II and III (as depicted in supplementary materials). Finally, deep learning models demonstrate substantial superiority over traditional approaches for lighter skin types (III and IV). However, for the most challenging skin types (V and VI), certain traditional methods perform comparably or even outperform some deep learning models, indicating skin tone-biased performance attributed to dataset imbalance.






To showcase the advantages of integrating the PhysFlow approach into rPPG data-driven models for addressing biased skin tone challenges, we assess our method on skin types V and VI, which represent the most challenging skin types, as mentioned previously. To mitigate the skin tone unbalancing effect, during training PhysFlow generates an augmented range of darker skin by conditioning each sequence with a luminance value between 0.15 and 0.40, which allows the model to generate and learn from a wider diversity of darker skin tones. Fig. <ref> illustrates a qualitative demonstration of the PhysFlow augmentation for dark skin tones, while ensuring the preservation of pulsatile information extracted from the facial region.

Table <ref> depicts the cross-dataset HR performance comparison among the three evaluated data-driven models for skin types V and VI in the MMPD dataset, with and without the integration of PhysFlow. The findings reveal a significant reduction in HR error, with the introduction of augmented dark skin tone videos resulting in a decrease in MAE ranging from 1 to 5 BPM, depending on the method. 
Skin type VI demonstrates a more substantial improvement compared to skin type V, attributed to its distinctiveness from the predominant skin type in the training dataset. While skin type V also experiences improvement, it is slightly less pronounced due to its proximity to the predominant skin type III in rPPG datasets. Additionally, there are variations in improvement among the adopted methods. For instance, the HR error for the EfficientPhys model is reduced by approximately 36 % for skin types V and VI, whereas the DPMN+TDM model shows reductions ranging from 13% to 22%. This can be explained since EfficientPhys was designed as a lightweight model for rPPG deployment which due to its limited characteristics is more sensitive to skin tone bias than PhysNet or DPMN+TDM. Nevertheless, after the PhysFlow data augmentation, the HR error reduction is observed across all tested models, and their performance aligns more closely with the tendency observed for skin type III in Table <ref>.








§ CONCLUSIONS AND FUTURE WORK



In this work, we introduce PhysFlow, a novel method for augmenting skin tone diversity in remote heart rate estimation.  PhysFlow utilizes conditional normalizing flows to disentangle skin tone information from other facial video content. Physflow is trained end-to-end, allowing simultaneous training of any supervised rPPG approach on original and augmented data during data generation. To train PhysFlow we use a novel bi-dimensional skin tone representation using CIELAB color space, offering adaptability to unbalanced rPPG datasets and skin tone variations. Our cross-dataset experiments on the MMPD dataset using three different data-driven models demonstrate the capability of PhysFlow for skin tone diversity augmenting in any supervised rPPG, showing how our approach significantly reduces heart rate estimation error, particularly in underrepresented skin tone categories, favoring equitable performance across different skin tones. Future work will focus on extending the data augmentation process to include the generation of new appearance data and modification of pulsatile information through a multi-modal conditional normalizing flow approach. 


§ ACKNOWLEDGMENTS

This work is partly supported by the eSCANFace project (PID2020-114083GB-I00) funded by the Spanish Ministry of Science and Innovation.





 Paper Title: Taming the Frequency Factory of Sinusoidal Networks.




Enhancing Deep Hedging of Options with Implied Volatility Surface Feedback InformationWe thank John Hull and Zissis Poulos for valuable feedback about this work. Gauthier is supported by the Natural Sciences and Engineering Research Council of Canada (NSERC, RGPIN-2019-04029), a professorship funded by HEC Montréal, and the HEC Montréal Foundation. Godin is funded by NSERC (RGPIN-2024-04593).
 
    [
     
August 5, 2024
=============================================================================================================================================================================================================================================================================================================================================================================================================




§ ABSTRACT 
This work investigates the structure and representation capacity of sinusoidal MLPs, which have recently shown promising results in encoding low-dimensional signals.
This success can be attributed to its smoothness and high representation capacity.
The first allows the use of the network's derivatives during training, enabling regularization. 
However, defining the architecture and initializing its parameters to achieve a desired capacity remains an empirical task. This work provides theoretical and experimental results justifying the capacity property of sinusoidal MLPs and offers control mechanisms for their initialization and training. We approach this from a Fourier series perspective and link the training with the model's spectrum.
Our analysis is based on a harmonic expansion of the sinusoidal MLP, which says that the composition of sinusoidal layers produces a large number of new frequencies expressed as integer linear combinations of the input frequencies (weights of the input layer). We use this novel identity to initialize the input neurons which work as a sampling in the signal spectrum. We also note that each hidden neuron produces the same frequencies with amplitudes completely determined by the hidden weights. Finally, we give an upper bound for these amplitudes, which results in a bounding scheme for the network's spectrum during training.







§ INTRODUCTION




Implicit neural representations (INRs) have emerged as a powerful method to represent media objects, such as images and surfaces.
They contrast with the classical techniques as they encode the data continuously on the parameters by mapping the coordinates to the signal values.
For example, in the case of an image, an INR f receives a point (x, y) and returns the  value f(x, y). In this case, the network's domain is low-dimensional, which differs from classical machine learning representations, where the input is a discrete image. Another difference is that we would like the INR to fit the input data as close as possible, as in the problem of approximating signals using radial basis functions.

As noted by <cit.>, since multilayer perceptrons (MLPs) activated by  cannot introduce high frequencies, they are not capable of representing detailed signals. 
To avoid getting stuck in a local minimum of the loss function where the INR does not contain high frequencies, they project the input coordinates onto a list of sines and cosines (Fourier feature mapping). 
This projection allows the INR to bypass the spectral bias <cit.> and represent detailed signals.

In parallel, <cit.> presented an initialization scheme for sinusoidal MLPs, which were considered difficult to train due to the initialization leading to undesirable local minima <cit.>. Their scheme ensures stability and convergence during training. 
An observation that partially justifies the representation power of a sinusoidal INR is that its first layer acts as a Fourier feature mapping <cit.>.

Several works used sinusoidal INRs to represent images <cit.>, surfaces <cit.>, animation <cit.>, and radiance fields <cit.>. They have empirically demonstrated that such INRs have a high capacity to represent fine details. However, they also show well-known limitations <cit.>, such as controlling the signal bandlimit and determining the appropriate parameters to efficiently represent a given signal.

This work considers an alternative approach for such problems through a novel trigonometric identity (Theorem <ref>) which resembles a Fourier series of a sinusoidal neuron.
We use this formula to understand the network's capacity by investigating how the layer's composition compacts information. 
In addition, we explore practical issues of initialization in sinusoidal INRs providing a reinterpretation of SIREN's scheme and ways of improving it.
For this, we note that the expansion of each sinusoidal neuron produces a large number of new frequencies in terms of the first layer weights (input frequencies). 
We also prove that the corresponding amplitudes are bounded by a term depending only on the INR's hidden weights (Theorem <ref>).
Finally, we apply this result to control the bandlimit of the network spectrum during training.
In summary, our contributions are:

    
  * We present a trigonometric identity that says that each sinusoidal neuron expands as a wide sum of sines with its new frequencies being an integer linear combination of the input frequencies. 
    The corresponding amplitudes are given by the hidden weights. 
    
  * We provide an upper bound for the amplitudes in the expansion of a sinusoidal neuron, which offers tools to control the resulting signal using the hidden weights.
    
  * We use the trigonometric identity to initialize sinusoidal INRs. First, we initialize the input neurons using the fact that they determine the possible INR spectrum.
    Then, we initialize the hidden weights, taking into account that they determine the corresponding amplitudes.
    
    
  * We use the amplitude's upper bound to control the bandlimit of a sinusoidal INR. For this, we design schemes to bound the hidden weights during training.
    
    These boundings, along with the initialization generate frequencies progressively around the input frequencies.








§ RELATED WORKS




Implicit neural representation (INRs). INR <cit.> is a trending topic in the machine learning community that is used to learn highly detailed signals in low-dimension domains.
Current INR architectures use Fourier feature mappings <cit.> or sinusoidal activation functions <cit.> to bypass the spectral bias <cit.> common in  multilayer perceptrons.
The high representation capacity of sinusoidal INRs has motivated their use to represent a wide range of media objects. Examples include audio <cit.>, images <cit.>, face morphing <cit.>, signed distance functions <cit.>, displacement fields <cit.>, surface animation <cit.>, multi-resolution signals <cit.>, among others.
Most of these works exploit the derivatives of sinusoidal INRs in the loss functions.


Initialization.
Considering sinusoidal activation functions in neural networks is a classical problem <cit.>; however, these INRs have been regarded as difficult to train <cit.>. <cit.> overcome this by presenting a specific initialization scheme that allows training sinusoidal INRs, avoiding instability and ensuring convergence.

Despite these advances, in practice, the initialization of such INRs remains an empirical task. In this work, inspired by Fourier series theory, we present a novel initialization method that allows us to train INRs with bandlimit. 
Recently, several works have addressed the representation problem of sinusoidal INRs from different perspectives. <cit.> approached this problem by observing that the first layer of a sinusoidal INR is similar to a Fourier feature mapping. 
Here, in addition to improving the initialization of sinusoidal INRs, we present a training scheme for bounding the spectra of these networks.




Control of spectrum.
One of the main drawbacks of sinusoidal INRs is the lack of frequency control. <cit.> addressed this issue by multiplying the input frequencies by a factor ω_0 to scale them to an appropriate range. However, even an appropriate choice of ω_0 could result in a noisy reconstruction since we have no control guarantees over the network spectrum.
We avoid the ω_0 problem by providing a novel initialization for the first layer. This is inspired by the fact that the frequencies generated by the network are determined by the input layer. Additionally, we provide a bounding scheme during training considering that the hidden layers define the frequency amplitudes.
Other works <cit.> employed multiplicative filter networks (MFNs) <cit.> to control the bandlimit by applying a filter over the spectrum. However, this strategy is prone to introducing reconstruction artifacts. It is important to note that MFNs are fundamentally different from neural networks as they are not modeled by the composition of sinusoidal layers, which is the focus of this work.




§ A FOURIER SERIES INTERPRETATION OF SINUSOIDAL INRS





This work addresses the problem of obtaining an efficient initialization and a training scheme with controlled bandlimit for the frequencies of sinusoidal INRs. This section presents the mathematical definitions and some novel formulas for approaching such tasks.








Sinusoidal INRs demonstrated a high representational capacity with only a few hidden layers <cit.>. 
Thus, to understand how the composition of its layers are compacting the information, we propose to investigate a sinusoidal INR f:^d→ consisting of a single hidden layer S:^m→^n, with m,n∈.
Specifically, we consider f to be expressed as f(x)= L∘S∘D(x),
with S(x) = sin(Wx + b), where W∈^n× m is the hidden matrix, and b∈^n the bias.
The input layer D(x)=sin(ωx+φ) projects the input x into a list of harmonics (input neurons) D_i(x)=sin(ω_i x+φ_i) with frequencies ω=(ω_1,…, ω_m)∈^m× d and shifts φ∈^m.
Finally, L(x)=Cx+d is a linear layer.
We show empirically that such architecture has enough capacity for our purposes.

We now present some properties of the sinusoidal layers, which play key roles in the representation, and give a reinterpretation of them in terms of the network parameters.

First, note that the hidden neurons h(x) := S∘D(x) are defined by composing the dictionary D with the hidden sinusoidal layer which results in a list with elements 
h_i(x)=sin(∑_j=1^mW_ijsin( ω_jx+φ_j)+b_i).



We now present a key result of this work, which states that we can linearize a hidden neuron by a sum of sines with frequencies and amplitudes completely determined by ω and .  

    Each neuron h_i of a sinusoidal INR has an amplitude-phase expansion of the form
    
    h_i(x) = 
                ∑_k∈^mα_k sin(β_k x + λ_k),    where   β_k = ∑_j k_j ω_j,   λ_k=∑_j k_jφ_j + b_i,

    and α_k = ∏_j=1^mJ_k_j(W_ij) is the product of the Bessel functions of the first kind.


The Bessel functions J_k appear in the Fourier series of sin(asin(x)) <cit.> and Theorem <ref> is a generalization of this result.
The complete proof adn details can be found in Appex <ref>, and Appedx  <ref> gives a generalization for more hidden layers.


[13]r0.42
     

    
    < g r a p h i c s >

    

    Comparison between INRs with zero and one hidden layers. Training with one hidden layer is significantly faster (50s vs 13m), uses half the parameters (33842 vs 67803), and produces higher quality results (34.4dB vs 21.1dB).
    

We list some consequences of (<ref>).
First, the layer composition introduces a wide number of frequencies β_k depending only on ω, and shifts given by the input shift φ and the hidden bias b.
Precisely, truncating the expansion by summing over k_∞≤ B∈, [k_∞ denotes max{k_1, …, k_m}.] implies that each neuron h_i could learn (2B+1)^m -1/2 non-null frequencies.
This frequency factory
 explains why composing sinusoidal layers may compact data information. Indeed, Fig <ref> shows that training with fewer parameters using a narrow sinusoidal INR with one hidden layer performs better than a wide INR with no hidden layer.
Additionally, we explore this factory to present an initialization of the INR's input neurons in Sec <ref>.

Secondly, note that the weights W fully determine the amplitude α_k of each harmonic sin( β_k x+λ_k) in the expansion. 
Additionally, we can derive the corresponding sine-cosine form of (<ref>),


    h_i(x) 
       =∑_k∈^mA_ksin(
            β_kx
        ) +B_kcos(
            β_kx
        ),    with   A_k=α_kcos(λ_k),    B_k=α_ksin(λ_k).

Note that the generated frequencies are independent of the index i, thus the hidden neurons share the same harmonics: sin(β_kx), cos(β_kx).
Since different linear combinations of the input frequencies may correspond to a single generated frequency, (<ref>) isn't (yet) the Fourier transform of the network's output. In other words, we could have two integer vectors k, l∈^m such that β_k=β_l. In Appex <ref>, we show how to aggregate those frequencies to obtain the final Fourier transform.

A natural issue regarding the frequency factory is the existence of B>0 such that α_k is small for k_∞≥ B.
We can use the formula of α_k to determine a bounding, showing that if W_∞<2 then the coefficients A_k and B_k decay exponentially as k_∞ grows.

  
    The magnitude of the amplitude α_k is bounded by
    ∏_j=1^m( |W_ij|2)^|k_j|1/|k_j|!.
    


Sec <ref> presents a bandlimit control mechanism during training based on Theorem <ref>.



§ INITIALIZATION AND FREQUENCY BOUNDING


This section presents an initialization and frequency control schemes for sinusoidal INRs.
Our initialization considers integer input frequencies (based on a Fourier series interpretation) and uses Theorem <ref> to study the spectral content of layer composition.
Then, we use Theorem <ref> to control the bandlimit of sinusoidal INRs during training.


 §.§ Initializing the input layer is a sampling in the spectral domain




[11]r0.17
     

     [b]0.17
         
    < g r a p h i c s >

         
     
        
        

First, we initialize ω with integer frequencies by setting ω_j ∈2π/p^d
and freezing them during training, where p∈^+ will be the INR's period.
This guarantees that the input neurons D are p-periodic.
Then, Theorem <ref> ensures that such periodicity is preserved by the layer activation. On the right, we show a comparison between SIREN and our initialization with p=2, both supervised only inside the red square (image from <cit.>).
This gives a proof of the empirical results about periodicity in <cit.>.
Additionally, (<ref>) implies that we can rewrite the INR as:




    f(x) =∑_k∈^mC,A_ksin(β_kx) + C, B_kcos(β_kx) + d





Since the generated frequencies β_k only depend on the frozen parameters ω, the training is responsible for learning the amplitudes of the sine-cosine series in (<ref>).








Assume that the ground truth has a bandlimit B.
First, note that we can sample ω directly on the cube [-B, B]^d, since (<ref>) indicates that the chosen frequencies appear in the spectrum of f. 

However, it also shows that the composition of layers produces an exponential number of new frequencies. These may extrapolate the cube
since they are integer combinations of the input frequencies.
Therefore, we propose to sample ω_j in 2π/p[-b, b]^d, with 0<b<B a bandlimit for the input frequencies. 
Indeed, as the hidden matrix  is initialized with small values (|W_ij|<2, see Sec <ref>), the magnitudes of A_k and B_k decrease as k_∞ increases.
This is a consequence of Theorem <ref>:



    |α_k| ≤∏_j=1^m( |W_ij|/2)/|k_j|!^|k_j|≤1/|k_1|!…|k_m|!  for   i=1,…,n.

Thus, the generated frequencies β_k with small k have more influence over the expansion (<ref>), mimicking  Fourier series theory.
This motivates us to initialize the input frequencies close to the origin; we found b=B/3 to work well with the experiments.
Fig <ref> shows that it is necessary, as higher b introduces noise.
However, in next section we avoid this problem by bounding the INR's spectrum.



Note that if ω_j is an input frequency, -ω_j is also present in the initialization. 

This is a consequence of sin(ω_j+φ_j)=cos(φ_j)sin(ω_j)-sin(φ_j)sin(-ω_j+π/2) being a linear combination of harmonics with frequencies ω_j, -ω_j.





Then, we only need to sample ω_j in half of the cube [-b, b]^d.
Also, from (<ref>) we have that the final bias d represents the amplitude of the frequency (0,0), hence we do not initialize it in ω.
Finally, since D()=sin(ω+φ), we initialize φ uniformly in [-π/2, π/2].












[11]r0.75
     
     
     

         
    < g r a p h i c s >

         

        INR spectrum during training. The input frequencies ω in green. During training new frequencies appear in a neighborhood of ω. The red dot indicates a high input frequency ω_j and the red arrows show the generated frequencies k_jω_j, k_j=1, 2, 3.
    

We now discuss an initialization of the frequencies ω∈^m× d.
First, note that (<ref>) indicates that only frequencies β_k with k small can have high amplitudes. Then we initialize ω such that it determines a sampling on the spectral domain.




Specifically, we split [-b, b]^d in regions L=[-l, l]^d of low and H=[-b, b]^d∖L of high frequencies, l∈(0,b).


When ω_j∈L, small variations in k_j results in a frequency near β_k. Indeed, Fig <ref>b shows how low frequencies (green square) introduce frequencies around the input frequencies (Fig <ref>a).
On the other hand, if ω_j∈H then β_k+e_j is far away from β_k; we observe this in Fig <ref>b-e, where ω_j is the red point and the arrows show the generated frequencies k_jω_j.














[8]r0.45


     [b]0.22
         
    < g r a p h i c s >

         
     
     [b]0.22
         
    < g r a p h i c s >

         
     
     

    Choosing ω as the cartesian product of the odd frequencies w/wo (1,0), (0,1).
    

When sampling frequencies in L, we include (1,0) and (0,1). This guarantees that any frequency could be generated and prevents the appearance of sub-periods (see Appex <ref>).
Fig <ref> shows an example in which this condition is violated, leading to an unappropriated reconstruction. 
For the higher frequencies in H, we sample in a grid-like fashion, uniformly covering the half square K. 








 §.§ Frequency bounding and hidden layer initialization






Sinusoidal INRs can represent detailed signals due to the exponential number of frequencies generated by the sine activation.
However, they could manifest as noise in the reconstruction.
Here, we use Theorem <ref> to introduce a bounding 
scheme to control the INR bandlimit during training.
[13]r0.55

     [b]0.55
         
    < g r a p h i c s >

     
     

    Training for 6000 epochs with b=30 and bounds c=0.1, 0.2, 0.5. The bound over the spectrum restricts the appearance of high frequencies.
    

Then, we use the relationship between the hidden matrix , generated frequencies β_k, and ω to define the bounding by blocks.
Next, we initialize  in a range defined by the bounds, improving performance.
Finally, we define a new architecture that allows us to learn the bounds.
Precisely,
using Theorem <ref>, we have that the amplitudes of the generated frequencies are limited by:

    |α_k|≤(_i_∞2)^k_11/∏_j=1^m|k_j|!.



This implies that if _∞≤2 then the amplitude of a generated frequency β_k rapidly decreases when the coordinates of k grow. 
Thus, forcing _∞≤2 during training reduces the appearance of  β_k with high |k|_∞ in the reconstruction.
Therefore, we use (<ref>) to define a bounding scheme that controls the INR bandlimit so that the generated frequencies saturate around small multiples of the input frequencies. 
It simply clamps the coefficients W_ij by a bound parameter c∈(0,2]. 

This procedure is applied at each epoch, enforcing _∞≤c.

Fig <ref> presents the results of bounding  with different bounds c.


We initialize  with values in [-c, c] using a normal distribution with mean 0 and standard deviation c/3.
Observe that 
SIREN <cit.> also controls the bandlimit but only at initialization, since the weights of  are sampled in (-√(6/m), √(6/m)). 
In other words, since _∞<2 (for m>1) at initialization, they do not introduce high frequencies at the beginning of the training. Additionally, we note a faster convergence when training with bounding using our initialization (see Appex <ref>).




In Sec <ref> we split the input frequencies ω in two sets: the low (ω_j_∞≤l) and high (ω_j_∞>l) frequencies.
Now, observe that each input frequency ω_j is associated with the j-column 
 of .
Then, by bounding the weights of  related to high frequencies, we restrict the amplitudes of β_k that could exceed the Nyquist limit B. Conversely, since low frequencies are used to span around ω, we consider a higher bounding for their columns.
Precisely, we define two parameters c_1 and c_2 to bound the weights W_ij corresponding to ω_j in L and H, respectively,





    ω_j_∞≤l →  W_ij=(W_ij, [-c_1,c_1]);     ω_j_∞>l →  W_ij=(W_ij, [-c_2,c_2]).





Analogously, we initialize  such that each column is normal distributed with mean 0 and standard deviation c_l3, with c_l the bound corresponding to that column.


Finally, we present a scheme to learn the bounding parameters c_j during training. For this, we bound each j-column W_*j of the hidden matrix W using a tanh activation followed by multiplication with a trainable bounding parameter c_j, that is, tanh(W_*j)c_j. Clearly, the resulting column is bounded by c_j. In other words, we are replacing the hidden layer 
sin(Wx + b) by





    sin(tanh(W)Cx + b)  whereC is a diagonal matrix with each jj-entry being c_j.





To prevent the bounds from getting too big, we use Theorem <ref> to define a regularization ℒ_reg=∑c_j.
Otherwise, the hidden weights virtually grow unbounded and the INR overfits as in the usual training.







§ ADDITIONAL EXPERIMENTS AND COMPARISONS






This section presents experiments regarding the initialization and bounding of frequencies of an INR f.
Our code is based on the MRNet <cit.> repository, using the images of the Kodak dataset resized to [-1,1]^2. 
Since our architecture is determined by the hidden matrix ∈^n× m, we use n and m to define f. 
We initialize the network using the scheme given in Sec <ref>.

We use Adam's optimizer <cit.> with a fixed learning rate 1e-4.
For the quantitative evaluations, we use the mean and standard deviation of the PSNR evaluated on the test set (10% random pixels).
Besides comparing the reconstruction quality of our INR f, we also evaluate the resulting analytical gradient ∇ f to show that our method adds high order regularity in the signal reconstruction. All experiments were performed in a GPU NVIDIA TITAN X (Pascal) with 12 GB.





 §.§ Initialization






[10]r0.5



    
    2-7
    1l|     2c|30%     2c|50%     2c70%                            
 1-7 l
                              Signal            Grad            Signal            Grad            1cSignal     1cGrad 
 21     35.2     26.9     35.2     26.6     35.0     26.0 
 42     35.1     26.3     34.8     24.6     33.8     22.2 
 64     34.2     25.2     33.3     21.6     30.8     15.0 
 
    

    Training an INR with 176K params. for 400 epochs. 
    Results indicate that it is better to sample more ω_j∈L and choose a smaller l.
    
    

First, we compare our ω initialization against the uniform distribution. We obtained a 4dB improvement of PNSR on average (with a std of 1.6dB). 
We also present an ablation over the parameter l which splits the square [-b, b]^ 2 in regions of low (L) and high (H) frequencies (see Sec <ref>).
Here we consider b=85 and l=21, 42, 64, 

and sample ω with 30%, 50%, and 70% of coordinates in H.
Table <ref> shows the performance of the INR, with the signal (grad.) mean std of 1.9dB (3.1dB).

[7]r0.5


     [b]0.5
         
    < g r a p h i c s >

     
     

    Reconstructions with large l present artifacts (ovefitting) on the 10% unsupervised points. 
    
    
    

This experiment considered all the images in the dataset.
Observe that all values of the row l=64 have less PSNR, showing that an INR with very high frequencies performs worse even for small percentages (30%). 
Similarly, the performance worsens when we increase the number of high frequencies (last column).



We now test the influence of high frequencies by training an INR of size m=360, n=512 with b=256 over 90% of the ground truth pixels. We vary the parameter l=60, 120, 220 that defines low frequencies.


Increasing the number of higher frequencies, the INR suffers from noise and overfitting. This can be noted in the reconstruction on the unsupervised pixels and gradients in Fig <ref>.



 §.§ Bounding




In Sec <ref> we presented a scheme to bound the INR spectrum.
Here, we show additional experiments to demonstrate that this improves the signal and has a great benefit over the unsupervised gradient reconstruction.
Then, we give a comparison between our fixed and trained bounding schemes.


[12]r0.5


         
    < g r a p h i c s >

    
    Comparison of training wo/w bounds. Observe that training using bounds preserves high order information of the signal.
    
    
    

Fig <ref> shows a comparison between wo/w our bounding scheme to reconstruct an image of size 512^2. To demonstrate the impact of our scheme, we sample the input frequencies using b=256 and l=100. 
Without bounding (left), we effectively reconstruct the signal at the supervised pixels (30.8 dB), but it results in a noisy gradient (15.8 dB). That is, we are overfitting the signal. On the other hand, when bounding the hidden weights, we observe significant improvements in both the signal (35.1 dB) and its gradient (27.8 dB).
Our bounding scheme acts like a filter in both signal and gradient reconstruction.

[12]r0.5


    
    4-7
                                                                2lsignal     2|lgrad 
 1lb     1lc_1     c_2     1|lNB       WB       1|lNB        WB   
 1l256      1l1.5        0.05      1|l32.6           34.8        1|l14.5          25.8    
 1l190      1l1.5       0.05       1|l34.0           35.9       1|l18.5           26.1  
 1l190      1l1.5       0.2       1|l34.0            34.4      1|l18.5           21.0    
 
    
    

        Training using our bounding scheme (WB) improves the signal and grad. reconstruction. The choice of c_2 greatly impacts the grad. fidelity. The std of the difference between each image signal (grad) quality wo/w bounding (NB) is 1.4dB (1.6dB) on average. Evaluations performed on the test set (10% of pixels).
    

We perform a quantitative comparison, varying the parameters b and c_2, to numerically test our scheme on the dataset. Table <ref> shows an improvement in the signal for all cases. Furthermore, the unsupervised gradient of the image is better learned when using bounds, showing an average increase of 9.6dB against the standard training.
Also, the last two lines of the table show the impact of the parameters c_1, c_2 over the signal and gradient reconstructions. 
We consider training with c_1>c_2 since we are prioritizing low input frequencies in the spectrum generation.


Otherwise, this may imply a generation of high frequencies, possibly bypassing the Nyquist limit and resulting in a worse gradient reconstruction.







[13]r0.69


         
    < g r a p h i c s >

    
    

    Comparison of training with fixed/learned bounds. Learning the bound of each frequency allows an adjustment of the signal spectrum resulting in better signal/gradient reconstructions. Each blue point in the diagram represents a pair (ω_j, c_j).
    
    
    

Next, we show how we can learn each bound c_j. 
As described in Sec <ref>, we can incorporate a learnable c_j for each j-column of the hidden matrix through a modified layer (<ref>). 
Fig <ref> shows a comparison between our fixed bound scheme using initial un-tuned c_1=1.0 and c_2=0.6 (left) and the learned bound scheme (middle). 


As expected, training lowers the bounds for high frequencies (H) while maintaining or increasing the bounds of low frequencies (L). Thus, we achieve better reconstruction in both the signal and gradient.
In Fig <ref> (right), we note that low frequencies (left of the green line) have significant bounds, showing their importance during training.
Conversely, the bounds of frequencies in H rapidly decrease as they grow higher.
These schemes allow us to initialize with higher frequencies than B/3, accelerating convergence.







 §.§ Comparisons


We first compare our method with SIREN, which requires choosing the frequency parameter b,[Denoted as ω_0 in <cit.>.] a critical and often empirical task. Our approach uses b to control the generation of new frequencies. To demonstrate this, we perform a comparison by varying b as well as the number of input (m) and hidden (n) neurons.

First, consider the case where we have no information about the bandlimit of the underlying signal and can only rely on the Nyquist limit B. Fig <ref> presents a comparison between SIREN (1^∘ row) and our method (2^∘ row) with b=B. Note that SIREN produces very noisy reconstructions during training, while ours learns faster and does not suffer from noise. This is due to our bounding scheme, which limits the generation of high frequencies and provides more control during training (Sec <ref>). In contrast, SIREN initializes its parameters such that the spectrum is initially bounded but does not maintain this guarantee during training.










Next, we consider a tuned b for SIREN to show that even under this condition, our method provides a better training scheme for sinusoidal INRs. Fig <ref> presents the comparison varying the number of input (m) and hidden (n) neurons. We consider the cases m<n (1^∘ row) and m>n (2^∘ row). Note that in both cases, our method performs better in the signal/gradient reconstructions. Also, we observe more robust training using our scheme, as seen in the loss functions in the 3^∘ column.













The experiments above showed that initializing SIREN with a high bandlimit (b= B) results in overfitting (Fig <ref>). Conversely, our method effectively learned the lower frequencies first and performed better than SIREN with the same configuration and a tuned b (Fig <ref>). Next, we numerically evaluate these experiments, providing a quantitative comparison with SIREN and BACON on the Kodak dataset.

We compare different architectures (m, n) and spectrum bandlimits b, evaluating the results in terms of signal and gradient PSNR over the supervised (Train) and unsupervised (Test) pixels. Table <ref> presents these evaluations. As expected, SIREN performs well with b=B/3. In this case, our method is perceptually similar to SIREN but improves gradient reconstruction and consistently outperforms BACON.

When b=171, our method demonstrates the best quality in both signal and gradient, showing greater robustness to spectral noise and superior performance even with b≥B/3. Finally, with b=256, our method surpasses SIREN in all cases. For BACON, our method achieves better signal reconstruction, with comparable gradients. However, the following experiment shows that BACON's reconstruction has visible defects in this scenario.

Note that we do not supervise the training with gradient information.


















Lastly, Fig <ref> presents a qualitative comparison between our bounding scheme and BACON.
We consider the three initial bandlimits b=85, 171, 256 as in Table <ref>.
We note that BACON truncates the spectrum in a similar way as the 0 hidden layer INR (see Fig <ref>), 
resulting in ringing artifacts.
Moreover, observe that it presents a loss of contrast as the bandlimit increases.
Conversely, ours operates as a soft filter providing better reconstructions in all cases.









§ CONCLUSIONS AND LIMITATIONS







We presented a study based on Fourier theory for sinusoidal INRs and used it to propose new schemes for both initialization and spectral control. 
Positively, our theoretical contributions generalize to different domains and aim to provide a better understanding of sinusoidal INRs for all applications. 
While we limited our experiments to images to demonstrate our theoretical claims; future works include using our schemes for other signals.
We also have focused our studies on a single hidden layer architecture leaving the investigation of deep architectures for a follow-up work.
In that sense, we are certainly considering future research aiming to extrapolate our results to general architectures and domains. Thus, we consider this work as a first step to unravel the powerful expressiveness of sinusoidal INRs.











plainnat











§ PROOFS OF THEOREMS



In this section, we prove Theorems <ref> and <ref>.
We consider the INR input to be 1D since the general case is analogous. 
Let f(x)= L∘S∘D(x) be a sinusoidal INR 
with S(x) = sin(Wx + b), where W∈^n× m is the hidden matrix, and b∈^n the bias.
The input layer D(x)=sin(ωx+φ) projects the input x into a list of harmonics with frequencies ω=(ω_1,…, ω_m)∈^m and shifts φ∈^m.
Finally, L is a linear layer.

The ith hidden neuron can be written as follows: 

    h_i(x)
    =sin(
        ∑_j=1^m W_ijsin(
            ω_jx+φ_j
        )+ b_i
    )


Note that, h_i receives a list of m input neurons D_j(x)=sin(ω_jx+φ_j) that are combined with the weights W_ij and activated by sin. 

Before presenting the expansion of h_i (Theorem <ref>), let us recall the (Fourier) expansion of a neuron with width 1 and no bias <cit.>:

    sin(asin(x))=∑_k∈ oddJ_k(a) sin(kx),  with   J_k(a)=∫_0^πcos(kt-asin(t)) dt.

The functions J_k are the Bessel functions of the first kind.
The expansion of h_i generalizes (<ref>). 







Its proof consists of verifying (<ref>) as well as a similar formula using the cosine as the activation function:

    cos(
        ∑_j=1^m W_ijsin(
            ω_jx+φ_j
        )+ b_i
    )
    = ∑_k∈^mα_kcos(
        k̇ωx +φ+ b_i
    ).

Now, we present the proof of Theorem <ref>, that gives the harmonic expansion (<ref>) of the neuron h_i,

    h_i(x) = 
                ∑_k∈^mα_k sin(β_k x + λ_k),   with  β_k = ∑_j k_j ω_j,  λ_k=∑_j k_jφ_j + b_i,  α_k=∏_j=1^mJ_k_j(W_ij).


For simplicity, we denote W_ij by a_j.

The proof is by induction in m. For the base case m=1, we prove sin(a sin(y)+b)=∑_k∈ J_k(a) sin(ky+b) with y=ω x+ φ.
For this, we use the expansion in (<ref>) and its cosine analogous expansion cos(asin(y))=∑ J_l(a) sin(ly), here the sum is over the even numbers. Thus, applying the trigonometric identity sin(a+b)=sin(a)cos(b)+cos(a)sin(b) we obtain:

    sin(a sin(y)+b)   =sin(a sin(y))cos(b)+cos(a sin(y))sin(b)
       =∑_k∈ oddJ_k(a) sin(ky)cos(b) + ∑_l∈ evenJ_l(a) cos(ly)sin(b)
       =∑_k∈ oddJ_k(a) sin(ky+b) + ∑_l∈ evenJ_l(a) sin(ly+b)
       =∑_k∈ J_k(a) sin(ky+b).

In the third equality we combined the formula sin(a)cos(b)=sin(a+b)+sin(a-b)/2 and the fact that J_-k(a)=(-1)^kJ_k(a) to rewrite the summations.
The proof of the formula using the cosine as an activation function is similar.

Assume that the formulas hold for m-1, with m>1, we prove (<ref>) holds for m (the induction step). Again, we denote y_j=ω_j x+ φ_j and b_i as b for simplicity.


    sin(∑_j=1^m a_jsin(y_j)+ b)   =sin(∑_j=1^m-1 a_jsin(y_j)+ b)cos(a_msin(y_m))
       +cos(∑_j=1^m-1 a_jsin(y_j)+ b)sin(a_msin(y_m))
        
       =∑_l∈^m-1,  k∈ evenα_lJ_k(a_m)sin(l̇y+ b)cos(ky_m)
       +∑_l∈^m-1,  k∈ oddα_lJ_k(a_m)cos(l̇y+ b)sin(ky_m)
       =∑_k∈^mα_ksin(k̇y+ b)

We use the induction hypothesis in the second equality and an argument similar to the one used in the base case to rewrite the harmonic sum. Again, the cosine activation function case is analogous.


<cit.> presented a similar formula for MLPs activated by polynomial functions.
While it is evident that the sine function can be approximated by a polynomial using Taylor series, our formula requires no approximations.
In addition to providing a simple proof, we also derive the analytical expressions for the amplitudes. These expressions enable us to compute upper bounds (Theorem <ref>) for the new frequencies α_k in terms of k and .
Next, we prove such inequality,

    α_k=∏_j=1^m( |W_ij|2)^|k_j|1/|k_j|!.



Again, we denote W_ij by a_j to simplify notation. Theorem <ref> says that
α_k=∏_j=1^mJ_k_j(a_j).
To estimate an upper bound for this number, we use the following inequality <cit.>, which gives an upper bound for the Bessel functions J_i(a).

    J_i(a)<(a/2)^i/i! ,    i>0,   a>0.

Observe that this inequality also holds for a<0 since J_i(-a)=J_i(a).
Therefore, replacing (<ref>) in ∏_j=1^mJ_k_j(a_j) and using J_-i(a)=J_i(a) results in the desired inequality.




§ TOWARDS COMPUTING THE FOURIER SERIES OF A PERIODIC INR



Here, we consider the INR f to have ^2 as domain, however, it could be easily extended to other domains.
Observe that a given frequency 𝔣 may appear several times as generated frequencies β_k=k^⊤ω with different coefficients k in the expansion (<ref>).
Here, we provide a characterization of all k∈^m such that k^⊤ω=2π/p𝔣, where p is the period 
of the sinusoidal network. This leads to an algorithm to compute the Fourier series of the periodic INR f.

To guarantee that a sinusoidal INR f initialized with ω= 2π/p[f_1, f_2]∈2π/p^m×2 can represent an arbitrary image, we need to determine when it can reconstruct a given frequency 𝔣∈^2.
This is equivalent to find a solution k∈^m to the system of Diophantine equations

    [f_1, f_2]^⊤k = 𝔣  for  𝔣∈^2.



Let B = U[f_1, f_2]^⊤V be the Smith normal form of [f_1, f_2]^⊤. Assuming e=U𝔣
such that e_i/B_ii∈ for i=1,2, then all integer solutions of (<ref>) have the form
V[[ e_1/B_11 e_2/B_22      l_1      ...    l_m-2 ]]^T
with l_1,..., l_m-2 arbitrary integers <cit.>.

Now, recall that we are considering ω_1=2π/p(1,0) and ω_2=2π/p(0,1), thus the Smith normal form and unimodular matrices U, V are given by,  

    B = [𝕀2 | 0_2× m-2],  U=I_2,   and  V = 
    [                                             𝕀2                                                -f_13    ...     -f_1m
    
    -f_23    ...     -f_2m;                                      0_m-2 × 2                                                                                          𝕀m-2 ].

Then e=𝔣 and the divisibility condition is satisfied since B_ii=1. Therefore, any frequency 𝔣∈^2 can be represented by f, and the integer solutions of (<ref>) are given by


    K(𝔣) = 
    [ 𝔣_1 - ∑_k=1^m-2 l_k f_1,k+2; 𝔣_2 - ∑_k=1^m-2 l_k f_2,k+2;                         l_1;                           ⋮;                       l_m-2 ]: l_1, ..., l_m-2∈.


Using K(𝔣) and (<ref>), we obtain the Fourier series of f by aggregating all coefficients associated with each frequency 𝔣∈^2. That is, the Fourier coefficients are given by 
A_𝔣=∑_k∈K(𝔣)C,A_k and B_𝔣=∑_k∈K(𝔣)C,B_k. 

To find a similar formula for the general case of ω∈^m× d, we must consider ω_i=e_i for i=1,...,d. Then, the aggregation of coefficients is analogous.



§ ON THE SUB-PERIODICITY OF SINUSOIDAL INRS


Observe that initializing a sinusoidal INR f using input neurons with period p implies that f is also periodic with period p (Theorem <ref>).
However, the initialization could force the INR to have sub-periods. In that case, it would never be able to fit a signal with fundamental period p.
We derive a condition to avoid such a problem.
First, recall that Theorem <ref> says that a hidden neuron 
h(x) =
sin(
    _i sin(
        ω+φ
    ) +b_i
), with ω = 2π/p[f_1, f_2] for some f_1, f_2∈^m, can be expressed as:

    h(x) = ∑_k∈^mα_ksin(β_kx + λ_k)

with frequencies and phase-shifts given by β_k=k^⊤ω and λ_k= k^⊤φ+b_i. Therefore,
β_kx = 2π/p(k, f_1x + k, f_2y).
Now, suppose that h has sub-period p/q in x-axis and p/s in y-axis (q,s∈^+), that is, h(x,y)=h(x+p/q, y+p/s).
However, (<ref>) implies

    h(x+p/q, y+p/s) = ∑_k∈^mα_ksin(β_kx + λ_k + 2πk, f_1/q+f_2/s).



Thus, since sin(x+2π k )=sin(x) for k∈, we have that Expansions <ref> and <ref> are equal only if k, f_1/q+f_2/s∈ for all k∈^m. Therefore, h (and consequently f) has sub-periods only if there exists q>1 or s>1 such that f_1/q+f_2/s∈^m.

An important consequence of our initialization is that the addition of the input frequencies (1,0), (0,1) to [f_1, f_2] prevents the appearance of sub-periods.
Indeed, we define the first elements of [f_1, f_2] as _11=1, _21=0, _12=0, and _22=1.
Hence, this implies that
_11/q+_21/s=1/q∈ if and only if q=1, and _12/q+_22/s=1/s∈ if and only if s=1.

Fig <ref> illustrates three cases where the choice of the input frequencies ω results in a bad reconstruction: 
(a) [f_1, f_2] with odd frequencies and period p=2; (b) [f_1, f_2] defined by the cartesian product of {1, 10, 100} and p=2, and (c)  [f_1, f_2] with odd frequencies and period p=3.
Fig <ref>c shows that in the case p=3 there is an overlap of copies generated by the periodicity.
This suggests an inconsistency between the selected period p and the resulting period of f.
Adding the frequencies (0,1) and (1,0) solves this problem, see Fig <ref> in the main text.





§ EXTENSION OF THEOREM 1 TO DEEPER NETWORKS


This section extends Theorem <ref> to multiple hidden layers. By recursively applying the argument from the proof of the theorem, we can establish the generalization using induction. We demonstrate the case of an INR with two hidden layers for simplicity, however, the general case is analogous.

Let f(x) = L∘S_1∘S_0∘D(x) with S_i(x) = sin(W^(i)x + b^(i)). Then, a neuron of the last hidden layer can be written as 
h(x) = sin(
        asin(
            W^(0)sin(
                ωx + φ
            ) + b^(0))+ b^(1)), where a is a row of the hidden matrix ^(1).
Observe that by denoting y^(0)=W^(0)sin(           ωx + φ) + b^(0) we can apply (<ref>) as in the case d=1 as follows:

    h(x) 
         = sin(
            ∑_i=1^n a_j sin(
                y_i^(0)) + b^(1))
         = ∑_l∈^nα_l^(1)sin(
                l̇y^(0) + b^(1))


where α^(1)_l=∏_i=1^nJ_l_i(a_i). When replacing the term y^(0) in the sine of the last expression, it can be written as a composition of sines:

    sin(
            l, y^(0) + b^(1))
            = sin( 
            ∑_j=1^ml, _*j^(0)sin(
                ω_jx+φ_j
                ) + l, b^(0) + b^(1)) 
        = ∑_k∈^mα_k^(0)sin(
                k, ωx + k, φ + l, b^(0) + b^(1))

Here, α^(0)_k=∏_j=1^mJ_k_j(l, _*j^(0)).
The first equality is obtained by substituting y^(0) and aggregating the terms appropriately, the second one is given, again, by (<ref>) used in the proof of Theorem <ref>. Then, denoting
α_k, l = α_l^(1)α_k^(0) and λ_k, l = k, φ + l, b^(0) + b^(1)
we obtain the desired equality:

    h(x)
         = ∑_(l,k)∈^n× mα_k, lsin(
             k, ωx + λ_k, l)

Observe that several properties drawn from the case of an INR with a single hidden layer hold for the general case. First, the amplitudes α_k, l depend only on the hidden matrices, and still product of Bessel functions. On the other hand, only the biases of the model influence the shifts λ_k, l. This may imply that not all biases are necessary during training.


We emphasize that adding more hidden layers doesn't introduce new frequencies, as they are, once again, defined by k, ω. Then, deep sinusoidal networks just fine tune the amplitudes of the generated frequencies, which are completely determined by the choice of ω.



§ ADDITIONAL EXPERIMENTS




 §.§ Initialization in the half cube

[9]r0.13
    

    
    < g r a p h i c s >

    
    

We showed that initializing a frequency ω_i implies that its negative -ω_i also appears in the spectrum.
Here, we use this fact to avoid sampling duplicated frequencies in [-b, b]^2 to obtain a better reconstruction.  
We test the capacity of two INRs with the same architecture m=102, n=512, but with different initializations for their frequencies ω (blue and white dots in Fig <ref>a) and ω' (white/green dots in Fig <ref>b):

    ω =[(k,l)|   k,l∈{1, 3, 4, 7, 10, 20}]  and  ω' =[(k,l)|   (k,l)∈ω with  k>0] ⊔η.


Where η are additional frequencies (in green) sampled in the Cartesian product of the list [0, 1, 2, 3, 4, 7, 10, 20]. 
Fig <ref>c shows the PSNR of the resulting networks with respect to the iterations.
The results show that adding the new frequencies η results in a slight but consistent improvement in the PSNR during training.




 §.§ Initialization of hidden layer


In Section <ref> we proposed an initialization for the hidden matrix . The following experiment shows that an initialization that considers the bounding limit for  may grant a faster convergence.
In Fig <ref>, we initialize the INRs of size m=416, n=1024 with bandlimit b=82 and bounds c_1=1.0, c_2=0.2. The INR trains during only 10 epochs to observe the speed of convergence and fits an image with resolution 1024^2.
Observe that the INRs that considered the bounds to initialize their hidden layer  (2^∘ row) present a better reconstruction. This can also be observed in the zoom-ins of images, where our method presents more details.






 §.§ Learned bounds

In Section <ref> we presented a novel scheme to bound the sinusoidal INR during training. Furthermore, we also proposed an architecture that allows us to learn those bounds during training. In this section, we analyze the coherence of the resulting bounds as we vary the signal bandlimit.

We define several INRs with size m=n=416, initial bounds with value 0.5 and bandlimits b = 34, 85, 171, 256. Then, we fit them to images with resolution 512^2 using our scheme of learnable bounds. Fig <ref> shows the bounds trained after 400 epochs. Note that all plots have a green, dashed line that divides frequencies into low (L) and high (H).





Observe that in Fig <ref>a the low frequencies are saturated around a height of 0.37, showing that their use is reduced from the initial 0.5. Conversely, the amplitude of frequencies in H increases as the frequency gets higher. This shows that the INR finds it necessary to increase the amplitudes of high frequencies for a good reconstruction. 
For the case when b=85 (Fig <ref>b), training started discriminating which of the low frequencies contributes more to the reconstruction; this is observed by the spreading of bounds. As for the high frequencies, we observe a more uniform behavior, which may indicate that all frequencies may be contributing the same amount of information during training. This is also an indicator that the parameter b=85 (B/3) is a good bandlimit to train the INR.

Now, in Fig <ref>c-d, we can observe even more extreme behaviors. Low frequencies exhibit multiple distinct bounds, while high frequencies are suppressed to prevent noise generation.
The patterns depicted in the figures align with our theory, confirming that we are effectively learning a bound that serves as a mechanism for spectral control.



 Paper Title: Automatic Die Studies for Ancient Numismatics.





 



Automatic Die Studies






C. Cornet et al.




Université Paris-Saclay, CEA, List, F-91120, Palaiseau, France

clement.cornet/romaric.besancon/herve.le-borgne@cea.fr Centre d’Études Alexandrines (UAR 3134 CNRS/Ifao), Alexandria, Egypt

heloise.aumaitre/thomas.faucher@cea.com.eg IRAMAT-CEB (UMR 5060, CNRS-Université d'Orléans), France

julien.olivier@bnf.fr


Automatic Die Studies for Ancient Numismatics
    Clément Cornet10009-0001-5829-9741 Héloïse Aumaître20009-0003-5850-5087 Romaric Besançon10000-0003-1331-5768 Julien Olivier30000-0003-1579-1311 Thomas Faucher10000-0003-1016-3340 Hervé Le Borgne20000-0003-0520-8436
    August 5, 2024
==========================================================================================================================================================================================================================




§ ABSTRACT 
Die studies are fundamental to quantifying ancient monetary production, providing insights into the relationship between coinage, politics, and history. The process requires tedious manual work, which limits the size of the corpora that can be studied. Few works have attempted to automate this task, and none have been properly released and evaluated from a computer vision perspective. We propose a fully automatic approach that introduces several innovations compared to previous methods. We rely on fast and robust local descriptors matching that is set automatically. Second, the core of our proposal is a clustering-based approach that uses an intrinsic metric (that does not need the ground truth labels) to determine its critical hyper-parameters. We validate the approach on two corpora of Greek coins, propose an automatic implementation and evaluation of previous baselines, and show that our approach significantly outperforms them.
  





§ INTRODUCTION


The quantification of ancient monetary production helps understand the mint outputs and the relationship between coinage, politics and history <cit.>. This quantification primarily relies on die studies, which compare each coin in a series (with the same types) to determine the number of engraved tools (the dies) used to strike coins. Using statistical models <cit.>, it is then possible to estimate the number of dies used for each series and, consequently, the volume of coin produced. However, comparing each coin to the others in a corpus is a long and tedious manual work, which grows quadratically 𝒪(N^2) with the size N of the corpus, limiting current dies studies to a few thousand coins. An automatic approach to conducting die studies would not only offer an appreciable gain of time but  also open up prospects for larger-scale studies.



The first attempt in that direction is CADS <cit.>, a semi-automatic approach which consists in extracting ORB local descriptors at keypoints <cit.> in each image, deriving a similarity matrix then performing hierarchical clustering to identify some groups that may correspond to the coins struck by the same die. However, the approach requires two manual operations, for filtering the keypoints and for determining the threshold used to form the final groups. Another promising approach for automated die studies is Riedones3D <cit.> but it requires 3D-scanning every coin beforehand. The need for a 3D-Scanner is self-limiting, as it would in practice be very tedious to deploy at large scale for the millions of coins available.

In this paper, we propose a fully automated method for conducting die studies on low-resolution RGB images. Our approach involves extracting robust and fast deep learning features from coins, which are then filtered using a Random Sample Consensus method that operates without requiring hyperparameter settings. Subsequently, the matches between coins are used to construct a graph, and the coins are clustered using a graph clustering algorithm. To determine the crucial hyperparameter for graph clustering, we propose to rely on the intrinsic structural consistency of potential partitions. As a result, this method operates entirely without human intervention.

Our approach is evaluated on two die studies and compared to an automated version of CADS. It exhibits significantly better performance, while remaining fast: a die study on thousands of coins can be achieved in a couple of hours, while it takes several weeks for historians of antiquity. 
The code and models are made publicly available[project page: https://cea-list-lasti.github.io/projects/studies/studies.htmlhttps://cea-list-lasti.github.io/projects/studies/studies.html], not only for the sake of reproducibility, but also to be used by numismatists for their future die studies.



§ RELATED WORKS




  
Automatic and semi-automatic Die Studies
While most automated coin analysis systems are focused on coin types recognition <cit.> and corpus analysis <cit.>, some works tackle the problem of automating die studies. The seminal contribution in this direction is CADS <cit.>. It consists in extracting keypoints descriptors for every coin using ORB <cit.> and finding the corresponding matches. From these, they construct a pairwise dissimilarity measure upon the descriptors, and perform an agglomerative clustering algorithm <cit.> using this measure. The clusters obtained correspond to the coins struck by a same die. However, CADS is semi-automatic, as it requires human intervention at preprocessing and clustering stages.
In a preprint <cit.>, Heinecke propose an automated (“unsupervised”) approach based on a similar pipeline as CADS, with Gaussian process keypoint extraction and a Bayesian distance microclustering algorithm <cit.>. However, their code is only partially released, which makes it hard to reproduce their results. Our approach differs from both these works, with the use of more efficient deep learning local features and a fully automatic clustering method. 

The Riedones3D approach <cit.> exhibits promising results for automated die studies, relying on 3D-scanning every coin of the corpus. It is nevertheless limited in practice since it requires to re-scan every corpus with 3D-scan, while our approach relies on the RGB images of the coins, which are easier to acquire and generally already produced by numismatists for their die studies. 




  
Coin recognition
Coin image classification and recognition is a quite well-established task in computer vision, historically addressed with traditional segmentation approaches <cit.>. More recently, Joshi proposed a hybrid approach combining SVM and neural networks to identify small parts of manufactured objects and applied it to Indian coin recognition <cit.>. The neural network is nevertheless a classical feed-forward network that processes handmade visual feature vectors, as an alternative to the SVM. In the same vein Fonov and Ksenia also used a two-layer feed-forward neural network with visual features designed by hand such as filtered version of the coin image <cit.>. A more elaborated approach was proposed <cit.> which relies on two convolutional networks that directly learn the features from the image in accordance with the deep learning principles. They proposed to pool the features extracted by both neural networks with a bilinear pooling layer and added an attention layer to emphasize the salient features. This method has been applied to an image dataset of the Roman Republican coins that contains more than 18,000 images and obtained good performances of coin recognition. More generally, computer tools can be used to assist numismatists in the processing and analysis of large corpora of coins semi-automatically <cit.>. Recognizing a coin image nevertheless differs from the task we address in this paper, since the recognition of the die that is common to several coins requires distinguishing finer details. 



  
Image Matching

The goal of image matching is to find matching keypoints from a pair of images of the same instance (object or building) taken with different conditions (luminosity, orientation...). Traditional image matching techniques use classic keypoints detectors <cit.> with independent descriptors <cit.>. The lack of link between keypoints detection and descriptions leading to limited performance, more recent approaches adopted a joint detection and description <cit.>. Learned feature matchers <cit.> have shown remarkable improvements both in terms of accuracy and robustness. In addition, some recent techniques have shown promising results by performing joint segmentation and matching <cit.>. Lately, complex network architectures lead to a massive increase in runtime, yet speed becomes an important objective for modern approaches <cit.>. 

However, the task we address differs from the image matching one. While the latter aims at finding the matches between two images of the same instance, we consider a pair of images of two different instances of coins. We aim at identifying latent common properties, namely the fact that they were struck by the same die. Although the point of view and the illuminating conditions are usually more regular than in traditional image matching tasks, the two instances differ due to differentiated wear and tear of both coins and dies.




§ METHOD

We adopt a general pipeline similar to the one proposed in CADS, that comprises two main stages (see Figure <ref>). First, we extract matches from every pair of images from a collection in order to build a pairwise similarity matrix. Then, we perform a clustering using this similarity matrix to get a partition corresponding to the dies.





 §.§ Coin (dis)similarities

Similarly to the way numismatists compare coins in die studies, we look for matching features from every pair of coins. Differing from previous works <cit.>, we use deep-learning matches, namely XFeat <cit.>. XFeat is an image matching technique that obtains remarkable results on image matching benchmarks <cit.>, while being extremely fast, yet able to satisfy the need of time-gain required for larger-scale die-studies. An important hyperparameter is top_k, the maximum number of regions to consider for matching. As XFeat is trained on SfM tasks such as relative camera pose estimation, retraining it on coins (all taken from the same position above coins) would not make sense. We thus use pretrained weights, learnt from a blend of Megadepth <cit.> and synthetically warped COCO <cit.>.

Most image matching techniques - including XFeat - are designed to find matching keypoints between two images from a same object or building, while being robust to changes in orientation or luminosity. Here, we analyze different coin instances, with the hypothesis that coins from the same die must have similar structures. We expect the keypoints detector to be robust with regard to the preservation state of coins, die wear or the way coins were struck. 

In some cases, in particular when matching images from different coin instances, we could have irrelevant matches: matches sometimes occur in the background of the image, or on wear traces, mostly on the edge of coins. CADS <cit.> tackles this problem by filtering out the edges of the coins. However, this approach requires a human intervention to select the coin region to consider, which is not viable on large datasets. Moreover, important keypoints could still be near the edges of a coin. Totally cutting them off could lead to a performance loss.

We use the RANSAC-based MAGSAC++ <cit.> for filtering inlier matches from outliers. Assuming noise σ follows an uniform distribution, it fits an homography estimation via Iteratively Reweighted Least Square. At each step, the weights for each sample are recomputed from the marginalized over σ likelihood of a sample being an inlier (σ-consensus). Once the model has converged, we keep the matches that are identified as inliers. MAGSAC++ is robust, and does not require a user-defined threshold over σ to consider a sample as an outlier.

The number of filtered matches between two coins can then be interpreted and handled as a similarity measure. Then, repeating the process for every pair of coins in a collection produces a pairwise similarity matrix M. We derive a pairwise dissimilarity matrix as D=max(M)-M, better suited for further analysis. Other decreasing functions were tested to derive D from M, without noting significant difference in our experimental results. 



 §.§ Adaptive Graph Label Propagation

We construct a graph 𝒢 which nodes correspond to each coin of the considered corpus. Two nodes are connected if the number of matches between corresponding coins exceeds a threshold τ. When τ is chosen appropriately, coins should be connected only if they were struck by the same die. This approach of using an unweighted graph aligns more closely with traditional die studies.

To obtain our final partition, we perform graph clustering (or community extraction) over 𝒢, using the Label Propagation Clustering algorithm <cit.>. This approach first assigns a unique label to each node. It then iteratively updates each node's label to the most common label among its neighbors, processing nodes in a random order until convergence.

To determine the optimal threshold τ^*, we compute the partitions corresponding to every possible value of τ. For each of them, we compute the Silhouette Coefficient <cit.> that does not require ground truth clustering and only evaluates the coherence of the inner structure of a partition, given a dissimilarity measure. For a sample x_i that is in the k^th cluster C_k, it assesses whether it is closer to the other samples of its own cluster, or to the next-nearest cluster. We consider the value

    s(x_i) = b(x_i) - a(x_i)/max(a(x_i), b(x_i))


With a(x_i)=1/|C_k|-1∑_x_j∈ C_k, x_j x_iD(x_i,x_j) the average distance to same-cluster samples, and b(x_i)=min_l k1/C_l∑_j∈ C_lD(x_i,x_j) the average distance to next-nearest cluster samples. The Silhouette Coefficient of a whole dataset is then the average of s(.) for all samples in the dataset. 

The Silhouette Coefficient ranges from -1 to 1, with scores around zero for overlapping clusters, and close to 1 for dense and separated clusters, which are properties that the optimal partition should satisfy. Finally, we keep the partition that has the highest Silhouette Coefficient. The resulting Adaptive Graph Label Propagation (AGLP) approach thus allows to determine the groups that correspond to each die without any manual hyperparameter setting.



§ EXPERIMENTS



 §.§ Baseline: a fully automatic CADS approach

In order to compare our approach to previous works, we reproduce the CADS model <cit.>, with some adaptations to make the model fully automated, since the original model requires manual operations:


    
  * CADS only considers matches within a user-defined circle, in order to filter out edges. Human intervention is needed because the optimal radius of the circle varies from one coin to another. Since this process is not suited for automation, we ignore this filtering step in our fully automatic version of the model;
    
  * CADS builds a hierarchy between coins with agglomerative nesting <cit.>, but does not automatically decide where to cut it to build a partition, leaving the decision to the user.

We propose a first baseline CADS-AG^* that reproduces the semi-automatic approach CADS with a human user able to find the best possible cut for the automatic clustering. The second baseline is actually automatic, using a HDBSCAN clustering from the pairwise distance matrix, with a minimum cluster size of 2 (CADS-HDBSCAN).




 §.§ Evaluation framework



  §.§.§ Datasets

We perform the evaluation on two collections of coins, with their associated die analysis : the Paphos <cit.> and Tanis 1986 <cit.> collections, with respectively 2484 and 295 coins. We consider the coin images at a resolution of 288 × 288 pixels for Paphos, and 480 × 512 for Tanis 1986. In <cit.>, the CADS model was evaluated on a subset of 200 coins of the Paphos collection.

Since only a small proportion of coins from Antiquity have been found to this day, coins follow uneven distributions across dies on both collections (Figure <ref>). For example, 249 out of 2484 coins from the Paphos collection (9.98%) are singletons - the only ones to correspond to their respective die.



Also note that we only consider the obverse of each coin (not the reverse). For manufacturing reasons (mostly for Greek coins), the reverse die used to break more often, leading to 1856 (74.7%) singletons on the Paphos collection.



  §.§.§ Metrics


We aim to find the partition of a collection separating coins upon dies. To evaluate the results, we use three standard external clustering validity measures, that evaluate the similarity between two partitions U and V. All those metrics range from zero to one and are maximized for similar partitions.

The Adjusted Rand Index (ARI) <cit.> is based on the Rand Index (RI) that evaluates the proportion of sample pairs that are correctly partitioned, adjusted for chance against unbalanced clusters.


    ARI = RI - E[RI]/max(RI) - E[RI]


Similarly, the Adjusted Mutual Information (AMI) <cit.> is an adjusted version of the Mutual Information (MI) in an Information Theory sense. In the general case, AMI and ARI are often close, but AMI is supposed to be more relevant when the ground truth is unbalanced <cit.>. Noting H(.) the Shannon Entropy, the AMI of two partitions U and V is defined by :


    AMI(U, V) = MI - E[MI]/mean(H(U), H(V)) - E[MI]


Lastly, the Fowlkes-Mallows Index (FMI) <cit.> is the geometric mean of pairwise Precision and Recall, and is generally more relevant with noisy data or unrelated partitions.
We also report pairwise Precision and Recall, since a significant difference between both could help explaining poor results in some cases.





 §.§ Performance on Die Studies







We present a comparison of our model with the fully automatic CADS model <cit.> on the datasets Paphos and Tanis (<ref>). We first observe that, in the CADS approach, the HDBSCAN clustering performs better than the best obtainable result with the agglomerative nesting algorithm that was used in <cit.>. We also note that our method clearly outperforms CADS over every evaluation measure on both datasets, with most metrics above 0.90.

While the performances are much higher than CADS, some differences remain in comparison to die studies conducted by humans. The fraction of dies that are correctly identified among all those that should have been identified is reflected by the recall. The fraction of dies correctly identified among all those that have been identified is reflected by the precision. Since the number of dies is usually quite small in practice, a manual inspection of the identified groups can be conducted, leading to improved precision. However, the errors in terms of recall remain problematic since fixing them requires to inspect the full collection, thus losing the benefits of automation.





 §.§ Ablation and Analysis





  §.§.§ On the use of Silhouette to find the optimal threshold






As outlined in Section <ref>, we use the Silhouette Coefficient to establish a threshold for the number of matches, which allows us to transform coin similarities into a graph, on which we apply the Label Propagation algorithm. In this section, we aim to experimentally validate the relevance of this choice.

To achieve this, we compute partitions for every possible threshold and plot their Adjusted Mutual Information (AMI), Adjusted Rand Index (ARI), and Fowlkes-Mallows Index (FMI). These metrics help us evaluate the quality of the resulting clusters compared to the ground truth. Additionally, we assess the internal coherence of the clusters using the Silhouette Coefficient. The results of this study on the Paphos dataset are presented in <ref>.



We observe that all the metrics have very similar variations through different thresholds and are correlated to the Silhouette Coefficient. More precisely, if we compute the Pearson correlations of AMI, ARI and FMI with the Silhouette Coefficient, we obtain respective correlations of 0.973, 0.951 and 0.952 and the threshold that maximizes the Silhouette also maximizes all the metrics. These experimental results are also verified on the Tanis dataset and confirm the relevance of the use of the Silhouette Coefficient to estimate automatically the best threshold used to create the graph.
























  §.§.§ Comparison with other image matching models


To extract the matches, we chose the XFeat model for its recognized efficiency and effectiveness. In this section, we compare this model with the RoMa model <cit.>, which obtains state-of-the-art results in numerous benchmarks <cit.> with the objective of estimating the fundamental matrix between two images. This model estimates a dense warp between two images of the same object and outputs pixel-level probabilities for its dense warp. We consider a match above a certain threshold t.



From the results presented in <ref>, using RoMa matches instead of XFeat causes severe clustering degradations for every tested confidence threshold t. Especially with t ≤ 0.9, pairwise recall is much lower than with XFeat, meaning that lots of coins from the same die are not considered connected in this case. With t=0.8, more than half of connected coins are from different dies. These results show that even though RoMa is state-of-the-art on SfM tasks, it is outperformed by XFeat for the task considered in this study, which consists in quantifying matches between 2 different instances. Lastly, RoMa is much more computationally expensive, thus longer to compute, taking 22 hours to compute with two GeForce GTX TITAN X GPUs on the Tanis collection (295 coins), while it took only 5 minutes with XFeat. Therefore, we do not report results obtained with RoMa on the Paphos collection since it would take weeks of computation, which would be as long as doing the work manually. Our (perfectible) implementation of CADS required 2 days of calculations for Paphos and around 5 hours with our approach.






  §.§.§ Comparison with other clustering algorithms





To cluster coins from a similarity matrix of matches, our proposed method uses graph clustering, namely Label Propagation. In this section, we test our approach against other graph clustering algorithms and against other clustering approaches, directly based on the dissimilarity matrix or after projecting data in a Euclidean Space.



  
Graph Clustering 



Concerning graph clustering algorithms, we test using the Louvain <cit.> and Leiden <cit.> algorithms. These methods aim at maximizing the graph modularity, hence we use the same method to build a graph from a similarity before clustering. We also test using the graph's connected components as cluster labels (noted “Connected” in <ref>).



  
Clustering from a dissimilarity matrix

Given M a matrix of pairwise matches between coins, we derive a dissimilarity matrix D = max(M) - M and perform hierarchical clustering upon D with HDBSCAN <cit.>. It builds a hierarchy between samples from their density without making any assumption on the structure of clusters. Furthermore, it is able to output uneven partitions with outliers, as die studies usually require. We use HDBSCAN with a minimum cluster size of 2 and use outliers as singletons. Output clusters therefore  have no minimum size.



  
Projection to a Euclidean Space















Another possibility is to project our coins into a Euclidean Space before performing clustering. We use UMAP <cit.>, a manifold learning technique suited for clustering <cit.> that aims at preserving local more than global structure of data. From the dissimilarity matrix D, it projects data into n dimensions, with n a hyperparameter. Then, we perform clustering with HDBSCAN over the projected coordinates to obtain a partition. In a similar fashion as the method used to build a graph (<ref>), the optimal number of dimensions n^* is determined using the Silhouette Coefficient of the partitions corresponding to every n in [2, 100]. Since UMAP is not fully deterministic, it could lead to variations between different iterations. We thus report the average of 5 runs in  <ref>.





















































A first observation can be made that the method using UMAP projection before HDBSCAN clustering in a Euclidean Space obtains the poorest clustering performances. Even though UMAP is supposedly suited for clustering, it makes an assumption on the “number of neighbors” a sample has. This approach is the one that is the less adapted to our use-case.


Then, clustering with HDBSCAN from a dissimilarity matrix obtains much higher results in every studied measure than projection methods. In particular, its large improvement upon the projection-based method assesses that clustering directly from a dissimilarity instead of using UMAP improves results for our case, contrary to the general case. Finally, graph-based techniques seem to offer the best clustering results on both datasets, with all of those methods reaching very similar performances on the Paphos collection. The high AMI, ARI and FMI obtained while using connected components as clusters labels seem to show that clustering performance issues more from graph construction than from graph clustering itself. This simple strategy obtains the highest results on Paphos. Yet, its sensitivity to erroneous connections as well as its lower results with Tanis should not make it a default choice.
On both collections, Label Propagation, Louvain and Leiden clustering obtain the same results, being the best solutions regarding AMI, ARI and FMI. Note that Label Propagation is the fastest of those three approaches, typically being 1.7× faster than Louvain, and 12.9× than Leiden when computing every partition needed to determine the optimal graph threshold (<ref>).








  §.§.§ Impact of Filtering matches


As not all matches from XFeat are kept, we can evaluate the importance of this filtering stage, both in terms of clustering performances on known datasets (<ref>) and in terms of hyperparameter stability, as it could be crucial on unknown datasets. 

The main hyperparameter used for computing XFeat matches is top_k, the maximum number of regions of the image to consider for matching (also note that a higher top_k induces a higher computational cost). In  <ref> we represent its importance towards AMI, with and without filtering. 



Two observations can be made from these results. First, filtering consistently increases clustering quality. Second, the AMI observes fewer variations through different top_k values when filtering matches (the difference between higher and lower AMI on the studied range). Thus, filtering reduces the sensitivity to top_k. 
It can also be noted that AMI stabilizes for top_k > 5000. For this reason we use a default top_k=5000 that should obtain optimal results. On a hypothetical unknown collection that would require top_k > 5000, filtering would greatly limit the negative impact of having a suboptimal value.





  §.§.§ Summary of Ablation Studies








To summarize the different experiments and ablations, <ref> enlightens the importance of the different aspects of our method.
The most severe degradation occurs when using RoMa matches instead of XFeat. 

Clustering from a graph using Label Propagation allows the highest scores regarding every studied metric on both datasets.
Furthermore, using MAGSAC++ to filter matches has more impact towards clustering performances when clustering from a graph than from a dissimilarity matrix.



































































§ CONCLUSION AND LIMITATIONS



We proposed a fully automated solution for performing die studies. We did this by using a fast and robust keypoint detector+descriptor 
to find matches between every pair of coins. We then construct an optimal graph using intrinsic clustering metrics, and perform simple graph clustering to extract a partition upon dies. Our solution significantly improves on the state-of-the-art, and does not require any human intervention in its process. 
While the errors of our approach in terms of precision (less than 10%) can be further fixed manually (see <ref>), those in terms of recall remain problematic since fixing them requires to inspect the full collection manually, thus losing the benefits of automation. For existing die studies, which may not be perfect, our approach may help numismatists to quickly refine the initial annotation by providing false positive and negative samples to be re-examined (<ref>).




Thanks to this automated approach which code is released, 
ancient historians will be able to carry out die studies on a much larger scale, and thus deepen our understanding of the ancient world. This will be particularly useful in the case of civilizations for which die studies could not be completely carried out due to the size of the available corpora, such as Athen, Aegina or Roman Empire. 
This may also make it possible to study collections from several coin workshops and make links that humans would have thought impossible or unlikely a priori. 
However, the two corpora considered in this paper can be considered as quite homogeneous, since all the coins are represented by RGB pictures with about the same size, illumination conditions, viewpoint and resolution. Other collections likely to be the subjects of die studies can be much more heterogeneous when the various coins have been captured by different historians (or individuals), at different times, with different photo equipment. In some cases, all we have is a scan of the negative of the original photo, or the one printed in a historical research article, or even the image of a (plaster) mold of the coins or dies. The proposed approach is likely to be much less effective in such cases, in particular because the keypoints detector currently used is adapted to natural RGB images. Matching such heterogeneous data is an ambitious challenge, but is necessary to address die studies in all their diversity.

Acknowledgment this work was partially funded by the Agence Nationale de la Recherche (ANR) for the STUDIES project  ANR-23-CE38-0014-02.


splncs04



 Paper Title: Adaptive Mix for Semi-Supervised Medical Image Segmentation.





label1,label2]Zhiqiang Shen
label1,label2]Peng Caocor1
label1,label2]Junming Su
label1,label2]Jinzhu Yang
label3]Osmar R. Zaiane
[cor1]Corresponding author. 
School of Computer Science and Engineering, Northeastern University, Shenyang, China.
E-mail: mailto:xxszqyy@gmail.comxxszqyy@gmail.com; mailto:caopeng@mail.neu.edu.cncaopeng@mail.neu.edu.cn.
[label1]organization=School of Computer Science and Engineering, Northeastern University,
            city=Shenyang,
            postcode=110819,
            country=China

[label2]organization=Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education,
            city=Shenyang,
            postcode=110819,
            country=China

[label3]organization=Alberta Machine Intelligence Institute, University of Alberta,
            city=Edmonton,
            state=Alberta,
            country=Canada



§ ABSTRACT 
Mix-up is a key technique for consistency regularization-based semi-supervised learning methods, generating strong-perturbed samples for strong-weak pseudo-supervision. 
Existing mix-up operations are performed either randomly or with predefined rules, such as replacing low-confidence patches with high-confidence ones. The former lacks control over the perturbation degree, leading to overfitting on randomly perturbed samples, while the latter tends to generate images with trivial perturbations, both of which limit the effectiveness of consistency learning.
This paper aims to answer the following question: How can image mix-up perturbation be adaptively performed during training?
To this end, we propose an Adaptive Mix algorithm (AdaMix) for image mix-up in a self-paced learning manner. 
Given that, in general, a model's performance gradually improves during training, AdaMix is equipped with a self-paced curriculum that, in the initial training stage, provides relatively simple perturbed samples and then gradually increases the difficulty of perturbed images by adaptively controlling the perturbation degree based on the model's learning state estimated by a self-paced regularize.
We develop three frameworks with our AdaMix, i.e., AdaMix-ST, AdaMix-MT, and AdaMix-CT, for semi-supervised medical image segmentation. 
Extensive experiments on three public datasets, including both 2D and 3D modalities, show that the proposed frameworks are capable of achieving superior performance. For example, compared with the state-of-the-art, AdaMix-CT achieves relative improvements of 2.62% in Dice similarity coefficient and 48.25% in average surface distance on the ACDC dataset with 10% labeled data.
The results demonstrate that mix-up operations with dynamically adjusted perturbation strength based on the segmentation model's state can significantly enhance the effectiveness of consistency regularization.
The code will be released at https://github.com/Senyh/AdaMix.



Medical Image SegmentationSelf-Paced Learning Semi-Supervised Learning Mix-Up






§ INTRODUCTION


Medical image segmentation for delineating tissues, organs, and lesions from different imaging modalities is critical for computer-aided clinical applications, such as prognosis and preoperative planning. Deep learning has greatly advanced the progress of medical image segmentation, relying on its powerful representation capability and the availability of large amounts of labeled data <cit.>. However, obtaining pixel-level annotations is laborious and expensive in medical images, as it requires expert knowledge. 
Semi-supervised learning (SSL) has the potential to alleviate the label scarcity problem by leveraging a large amount of unlabeled data <cit.>. 







Consistency regularization, one of the most widely-used SSL paradigms, encourages model predictions to be invariant to input perturbations, aiming to sufficiently explore supervision information from unlabeled data <cit.>.
Building upon the idea of consistency regularization, strong-weak pseudo supervision <cit.> aims to enforce consistency between the predicted pseudo labels of strongly and weakly perturbed unlabeled images, thereby extracting additional supervisory signals from unlabeled data <cit.>. 
The key challenge of strong-weak pseudo supervision is determining appropriate strong-perturbation strategies for generating strong-perturbed images. Improper perturbations cannot match the training model's ability to obtain supervision signals from unlabeled data, misleading the training process <cit.>. 
Recently, CutMix[CutMix is formulated as x̅ = x_o × m + x_a × (1 - m) and y̅ = y_o × m + y_a × (1 - m), where x_o / x_a denotes an original/auxiliary image,  y_o / y_a refers to the corresponding (pseudo) label, and m is a randomly generated mask. x̅ / y̅ represents the mixed image / (pseudo) label.]<cit.>, incorporated with image color space transformation, has been widely used to form the strong perturbations <cit.>.
As illustrated in Fig. <ref>(b), CutMix randomly mixes image patches between an original image and an auxiliary image to synthesize a strong-perturbed example. 
However, such a random mix operation with uncontrollable perturbation degree leads to an oscillating and unstable consistency regularization process [Fig. <ref>(f)], where the model is prone to over-fit to the randomly perturbed examples, resulting in unsatisfactory generalization on the validation set [Fig. <ref>(g)].
To address the instability of the random mix-up operation, our previous conference version, UMix <cit.> [Fig. <ref>(c)] synthesizes new samples with more high-confidence regions, enabling the model to produce high-quality predictions for these new samples. 
As shown in Fig. <ref>(f-g), the model trained with UMix samples demonstrates smoother and lower training losses but achieves inferior validation and test performance due to the limited effectiveness of consistency regularization. 
Furthermore, we investigated the inverse UMix (I-UMix) strategy [Fig. <ref>(d)]. 
Since this operation yields new samples with overly strong perturbations, the model shows severe oscillation in the training loss and obtains poor validation performance [Fig. <ref>(f-g)].
It can be observed that random perturbations tend to be uncontrollable, while mix-up operations with fixed rules result in trivial or overly strong perturbations, both of which limit the effectiveness of consistency regularization.
Therefore, developing an adaptive perturbation/augmentation strategy for semi-supervised medical image segmentation is highly desirable.

Conceptually, a model's performance should vary dynamically with the number of training epochs increasing, thereby an appropriate mix-up operation should also dynamically adjust the perturbation degree based on the state of the segmentation model's capability.
Given this assumption, we propose Adaptive Mix (AdaMix), a novel dynamical and learnable perturbation algorithm that adaptively performs image mix-up based on a model's learning state, for better achieving consistency regularization in semi-supervised medical image segmentation. 
Our key idea lies in introducing the concept of self-paced learning <cit.> into the design of the mix-up algorithm and providing more reasonable perturbed images for strong-weak pseudo supervision with gradually increasing sample complexity, as depicted in Fig. <ref>(e).
The mix-up operation with varying degrees of perturbation leads to produce perturbed images with different levels of sample complexity.
An image that contains more high-confidence regions is considered a simpler sample by a segmentation model, as it tends to produce more accurate predictions from such an image.
Mixing with more high-confidence patches results in simpler perturbed images (with a weaker perturbation strength) and vice versa.
More specifically, based on the training loss, AdaMix generates 1) a self-paced mask for determining whether an image is mixed with high-confidence patches or low-confidence ones, and simultaneously provides 2) a self-paced weight for determining the number of mix-up patches.
One can observe from Fig. <ref>(f-g) that, compared with the random and fixed counterparts (i.e., CutMix <cit.> and UMix <cit.>),  AdaMix is capable of maintaining stable converging ability during the training process and achieving better generalization in the validation and test phase.
We empirically demonstrate that AdaMix can be seamlessly integrated into self-training, mean-teacher, and co-training (i.e., FixMatch <cit.>, MT <cit.>, and CPS <cit.>) SSL frameworks, resulting in AdaMix-ST, AdaMix-MT, and AdaMix-CT frameworks, respectively, which achieve state-of-the-art performance in semi-supervised medical image segmentation.
We evaluated the proposed frameworks on three public medical image segmentation datasets, involving 2D and 3D modalities. The results demonstrate the effectiveness of each component of our method and its superiority over the state-of-the-art. For instance, AdaMix-CT obtains 88.19% and 89.83% in terms of Dice Similarity Coefficient (DSC), outperforming MC-Net+ <cit.> by 2.09 % and 1.32% on the ACDC datasets under 10% and 20% labeled data, respectively.

In summary, our main contributions are:


  * New Problem: We investigate the problem of existing mix-up methods: they tend to generate perturbed images with uncontrollable, trivial, or overly strong perturbation degrees, which seriously affect consistency regularization and result in imprecise decision boundaries, particularly in heterogeneous lesion segmentation scenarios. 

  * New method: We propose a novel algorithm, AdaMix, for image mix-up in a self-paced manner. To the best of our knowledge, this is the first work to adaptively perform mix perturbation based on a model's state, where the segmentation model and the mix module collaboratively learn from each other. Furthermore, AdaMix is highly flexible and can be seamlessly integrated into self-training, mean-teacher, and co-training SSL paradigms.

  * New findings: 
We provide new insights: incorporating high-confidence patches into images generates simpler samples, which benefits the segmentation model during the initial training stage. 
In contrast, the low-confidence regions, which typically appear at segmentation boundaries, force the segmentation model to focus on these regions and enhance the model’s discriminative capability when mixed into images in the later training stages. 
We also found that, for consistency regularization, perturbation strategies are more important than learning paradigms (e.g., self-training, mean-teacher, and co-training).





§ RELATED WORK





 §.§ Semi-Supervised Learning


Semi-supervised learning (SSL) is an effective learning strategy that addresses the issue of limited labeled data by utilizing a large amount of unlabeled data <cit.>. 
Based on the design of the regularization term to leverage unlabeled data, many SSL algorithms have been proposed, such as adversarial learning <cit.>, pseudo labeling <cit.>, consistency regularization <cit.>. 
Recently, state-of-the-art SSL approaches have incorporated the idea of strong-weak pseudo supervision <cit.> as the primary strategy of their frameworks. According to the learning paradigms, these approaches typically fall into three branches: self-training-based 
<cit.>, 
mean-teacher-based <cit.> and co-training-based frameworks <cit.>. 
Similar to semi-supervised image classification tasks, the line of semi-supervised segmentation methods also can be divided into three branches: self-training-based <cit.>, 
mean-teacher-based <cit.>,
and co-training-based methods 
<cit.>.
The self-training paradigm involves a single model that generates pseudo labels for training the model per se <cit.>.
Mean-Teacher <cit.> can be considered an intermediate approach between self-training and co-training. It involves a teacher and a student model, where pseudo-supervision is conducted from the teacher to the student.
A representative co-training paradigm, presented in CPS <cit.>, comprises two (student) models that are cross-supervised by each other.
We found that perturbation strategies on unlabeled data are more important than a specific learning paradigm. We propose a novel image mix-up algorithm that can be seamlessly applied to these paradigms and achieves state-of-the-art performance in semi-supervised medical image segmentation.



 §.§ Mix-up Augmentation


Mix-up operations can be regarded as a line of data augmentation approaches in the form of blending two images and their corresponding labels to synthesize a new example.
A series of mix-up strategies were initially proposed in supervised learning to generate new samples via randomly mixing images in image-level <cit.>, feature-level <cit.>, and region-level <cit.>.
Later, additional information, such as saliency maps <cit.>, was incorporated into the mix-up algorithms.
Recently, image mix-up has been applied to perform perturbations for consistency regularization in a fully random manner <cit.> or according to predefined rules, e.g., replacing low-confidence patches with high-confidence ones guided by predictive uncertainty/confidence <cit.>.
However, the degree of random perturbations is uncontrollable, leading to overfitting to strongly perturbed samples, while the degree of fixed mix-up operations is trivial, both of which limit the effectiveness of consistency regularization. 
To solve this issue, we propose AdaMix which adaptively performs image mix-up perturbation in a self-paced learning manner for consistency regularization in semi-supervised medical image segmentation.
 




§ METHOD




 §.§ Problem statement


Before delving into the proposed method, we first provide the notations that will be used subsequently. The training set 𝒟 = {𝒟^ℒ, 𝒟^𝒰} contains a labeled set 𝒟^ℒ = {(x^l_i, y^l_i)_i=1^N^L} and an unlabeled set 𝒟^𝒰 = {(x^u_j)_j=1^N^U}, where x^l_i/x^u_j denotes the i_th/j_th labeled/unlabeled image, y^l_i is the ground truth of the labeled image, and N^L and N^U (N^U >> N^L) are the numbers of labeled and unlabeled samples. Given the training data 𝒟, semi-supervised semantic segmentation aims to learn a model f(·; θ) performing well on unseen test sets 𝒯.









 §.§ Overview


As discussed in Section <ref>, the random region mix-up <cit.> does not allow for adaptively controlling perturbation degree, leading to the learning model overfitting to randomly perturbed examples. Meanwhile, the confidence-guided mix <cit.>, which utilizes a fixed mix-up rule that replaces low-confidence patches with high-confidence ones, produces mixed images with trivial perturbation degrees, thereby limiting the effect of consistency regularization. In contrast, replacing high-confidence regions with low-confidence regions may generate new samples with overly strong perturbations, making it difficult for the model to adapt to these new samples. 
We pinpoint the question: How can image mix-up augmentation be adaptively performed during training? 
To this end, we propose Adaptive Mix (AdaMix), a novel augmentation algorithm for adaptive image mix-up based on a model's learning state. 
Building upon AdaMix, we construct an Adaptive Mix framework, where AdaMix acts as a plug-in-play module and can be seamlessly integrated into the self-training, mean-teacher, and co-training paradigms for semi-supervised medical image segmentation. 
In the following, we first specify the Self-Pacecd Mix framework [Fig. <ref>] and then collaborate on the Self-Pacecd Mix algorithm [Fig. <ref>].



   
  
 






 §.§ Adaptive Mix Framework


As illustrated in Fig. <ref>, the AdaMix framework includes two components: 1) the AdaMix algorithm for generating perturbed examples, and a semi-supervised learning scheme for conducting supervision on both labeled and unlabeled images.
According to the manners of strong-weak pseudo supervision <cit.>, semi-supervised learning can be realized through three paradigms: 1) self-training, 2) mean-teacher, and 3) co-training, in which FixMatch <cit.>, Mean-Teacher <cit.>, and CPS <cit.> are the corresponding representation frameworks, respectively. 

Formally, these paradigms can be formulated as follows.
Given (x^l_i, y^l_i) ∈𝒟^ℒ, x^u_j ∈𝒟^𝒰 and its corresponding pseudo label ỹ^u_j with the confidence map p̃^u_j, the semi-supervised learning objective can be defined as:

    J = min_θℒ_seg(f(x^l_i, θ), y^l_i)_ℒ_s + 𝕀_{p̃^u_j ≥τ}(p̃^u_j)ℒ_seg(f(Γ(x^u_i), θ), ỹ^u_j)_ℒ_u

where the first and second terms denote the supervised loss ℒ_s and the unsupervised loss ℒ_u, respectively. ℒ_seg(·, ·) is an image segmentation criterion, f(·, θ) denotes a segmentation model with learnable parameters θ, τ refers to a confidence threshold above which a pixel-level pseudo-label is retained for pseudo supervision. 
Γ(·) represents a strong perturbation/augmentation, which is the key to the efficacy of consistency regularization.

Self-Training involves a single model f(·, θ), which generates pseudo labels from unlabeled data for training of the model itself. A pseudo label is generated by: ỹ^u_j = maxSoftMax(f(Γ'(x^u_i), θ)), where SoftMax (·) denotes the SoftMax function that converts predictive logits into probabilities and Γ'(·) represents a weak perturbation (w.r.t. the strong perturbation Γ(·)) [Following the idea of strong-weak pseudo supervision <cit.>, in practice, Γ'(·) refers to 'weak' perturbations, generally consisting of only geometric augmentations, while Γ(·) refers to 'strong' augmentations, which include geometric and pixel degree augmentations, as well as mix-up operations.].







Mean-Teacher includes a student model f(·, θ) and a teacher f(·, θ̃) model, where the parameters θ̃ of the teacher model is updated by an exponential moving average (EMA) of the student's parameters θ in each iteration t: θ̃_t = αθ̃_t-1 + (1-α) θ_t. Pseudo labels are generated by the teacher model: ỹ^u_j) = maxSoftMax(f(Γ'(x^u_i), θ̃)).

Co-Training involves two (student) models, f(·, θ_1) and f(·, θ_2), which are cross-supervised from each other on unlabeled data. The pseudo labels yielded by one model are leveraged to supervise another model, i.e., ỹ^u_j[1/2]) = maxSoftMax(f(Γ'(x^u_i), θ_[1/2])).


Building upon these paradigms and the strong-weak pseudo supervision, we develop three frameworks with our AdaMix, i.e., AdaMix-ST, AdaMix-MT, and AdaMix-CT, for semi-supervised medical image segmentation. 
These instantiated frameworks employ Eq. <ref> as their learning objective and exploit AdaMix as the strong perturbation Γ(·) for consistency regularization. The training procedure of these methods is detailed in Algorithm <ref> (Note that, without loss of generality, we take AdaMix-ST as an example for illustration convenience).
We will show in Section <ref> that the proposed frameworks archive state-of-the-art performance on various medical image segmentation tasks.









 §.§ Adaptive Mix algorithm


As illustrated in Fig. <ref>, AdaMix is equipped with a self-paced curriculum that, at each training iteration, provides a self-paced mask and a self-paced weight to determine the mix-up rule and the number of mix-up patches, respectively, based on the model's learning state.
In our framework, AdaMix is applied to both labeled and unlabeled data. For convenience, we illustrate the algorithm details by taking AdaMix on unlabeled data as an example in the following. A detailed description of AdaMix on unlabeled data is provided in Algorithm <ref>.  Note that the only difference between AdaMix on labeled and unlabeled data is that the former uses ground truth segmentation labels, while the latter utilizes pseudo labels.

Given an unlabeled original image x^u_j, an unlabeled auxiliary image x^u_jj, and their corresponding pseudo labels ỹ^u_j, ỹ^u_jj, we first obtain their confidence maps by z^u_j = max(SoftMax(f(x^u_j, θ))) and z^u_jj = max(SoftMax(f(x^u_jj, θ))), respectively.
Meanwhile, we calculate the self-paced proxy loss ℒ̅ = ℒ̅_o + ℒ̅_a, where ℒ̅_o = ℒ_seg(f(x^u_j, θ), ỹ^u_j) and ℒ̅_a = ℒ_seg(f(x^u_jj, θ), ỹ^u_jj) denote the segmentation loss of the original image-pseudo label pair and the auxiliary image-pseudo label pair, respectively. 
Based on the self-paced proxy loss ℒ̅, we can define the self-paced mask and self-paced weight as follows.



  
Self-Paced Mask
Self-paced mask m ∈{0, 1} is a binary mask used to determine the mix-up rule r per image. Specifically, when m = 0, low-confidence patches are replaced with high-confidence patches L ← H; when m = 1, high-confidence patches are replaced with low-confidence patches H ← L:

    r = {    H ← L, if  m = 1
        L ← H, if  m = 0
    .

where H/L represents a set of high-confidence/low-confidence patches, and m is obtained by solving the following objective function:

    m^*(ℒ̅, λ) = min_m∈0,1mℒ̅ + h(m, λ)

where λ denotes the age parameter for controlling the self-paced learning pace. In general, a model's performance gradually improves as training progresses. Based on this, we employ a time-dependent Gaussian ramp-up function to gradually increase the age parameter, i.e., λ(t) = exp[-5(1-t/t_m)^2], where t denotes the current iteration and t_m is the maximum iteration in the training process.
h(m, λ) refers to a self-paced regularizer <cit.>. 
Considering the binary property of m, we employ a hard self-paced regularizer, i.e.,  h(m, λ) = -λ m, thereby obtaining the corresponding solution for m:

    m^* = {    1, if ℒ̅ < λ
        0, otherwise
    .




  
Self-Paced Weight
Self-paced weight v ∈ [0, 1] determines the number of mix-up patches n per image, defined as:
n = v × K
where K is a hyperparameter denoting the maximum number of mix-up patches. Similar to the calculation of the self-paced mask, v is obtained by:

    v^*(ℒ̅, λ) = min_v∈[0,1]vℒ̅ + g(v, λ)

where g(v, λ) = λ(1/2v^2 - v) is a linear self-paced regularizer. 
By solving Eq. <ref>, we obtain the solution for v:

    v^* =  1 - ℒ̅/λ


Finally, based on the confidence maps z^u_j and z^u_jj, the mix-up rule r, and the number of mix-up patches n, we perform patch mix-up with patch size S to derive an AdaMix perturbed example (x̅^u_j, y̅^u_j). Please refer to Algorithm <ref> for a detailed description of the procedure above.





§ EXPERIMENTS AND RESULTS





 §.§ Datasets


ACDC[https://www.creatis.insa-lyon.fr/Challenge/acdc/https://www.creatis.insa-lyon.fr/Challenge/acdc/] contains 200 short-axis cine-MRIs from 100 subjects, and the corresponding annotations with the left ventricle, right ventricle, and myocardium labels. Following <cit.>, we split the dataset into the new training, validation, and testing sets, respectively including 70, 10, and 20 patients’ data.

LA[http://atriaseg2018.cardiacatlas.org/http://atriaseg2018.cardiacatlas.org/] consists of 100 3D gadolinium-enhanced magnetic resonance scans and left atrial segmentation ground truths. Following <cit.>, we divide the 100 scans into 80 samples for training and 20 samples for evaluation.

ISIC[https://challenge.isic-archive.com/https://challenge.isic-archive.com/] includes 2594 dermoscopy images and the corresponding skin lesion annotations. We divide the entire dataset in a 7:1:2 ratio, resulting in 1815 images for training, 260 images for validation, and 519 images for testing.



 §.§ Implementation Details




Experimental environment: 
Hardware: NVIDIA A40 GPU with 48G GPU memory.
Software: Python 3.8, PyTorch <cit.> 1.11.0, CUDA 11.3.
We utilize AdamW <cit.> as the optimizer with a fixed learning rate of 1e-4. The training time is set to 100 epochs for the ACDC and ISIC datasets and 1000 epochs for the LA dataset.

Framework:
We employ U-Net <cit.>/V-Net <cit.> as the supervised baseline architecture for 2D/3D image segmentation, respectively. 
We utilize FixMatch <cit.>, MT <cit.>, and CPS <cit.> as the base frameworks for self-training, mean-teacher, and co-training, respectively. These frameworks build upon the concept of strong-weak pseudo supervision <cit.> to leverage unlabeled data.
We set the confidence threshold τ = 0.95 following FixMatch <cit.>.
For the hyperparameters of AdaMix, we set the mix-up patch size S = 32 and the maximum number of mix-up patches K = 16.
Detailed analysis for these hyperparameters is provided in Section <ref>.
In our experiments, we adopt a combination of cross-entropy and dice loss <cit.> as the segmentation criterion.



 









Data:
In the 2D image segmentation tasks, all images are resized to 256 × 256 for inference, while the outputs are recovered to the original size for evaluation. For 3D segmentation, we randomly crop 80 × 112 × 112   (Depth × Height × Width) patches for training and iteratively crop patches using a sliding window strategy to obtain the final segmentation mask for evaluation. 

Evaluation metrics. Dice similarity coefficient (DSC), Jaccard, average surface distance (ASD), and 95% Hausdorff distance (95HD) are employed to estimate the segmentation performance in our experiments.




 §.§ Comparison with State of the Arts


We extensively compared our proposed AdaMix frameworks with state-of-the-art semi-supervised segmentation methods, which can be divided into four groups based on their perturbation strategies: 
1)  Multi-task perturbation (using signed distance maps <cit.> or multi-scale features <cit.>): SASSNet <cit.>, URPC <cit.>, and DTC <cit.>; 
2) Noise perturbation: UA-MT <cit.>, MC-Net <cit.>, MC-Net+ <cit.>; 
3) CutMix perturbation: FixMatch <cit.>, MT <cit.>, CPS <cit.>;
and 4) UMix and I-UMix perturbations: UCMT <cit.>, and ABD <cit.>. 










  §.§.§ Results on ACDC

Table <ref> shows the results of our three AdaMix frameworks (i.e., AdaMix-ST, AdaMix-MT, and AdaMix-CT) and other semi-supervised segmentation methods on the ACDC dataset with 10% and 20% labeled data. All methods are evaluated on the average segmentation performance of left ventricular (LV) cavity, right ventricular (RV) cavity, and myocardium (MYO). 
The proposed AdaMix frameworks consistently outperform other approaches among all the metrics.
Specifically, AdaMix-CT achieves the best performance of 89.19% and 89.83% in terms of DSC under 10% and 20% labeled data, respectively. Concerning the boundary-sensitive metrics 95HD and ASD, with the help of AdaMix, AdaMix-CT obtains the best values of 2.46 and 0.74 for the 10% labeled data. 
This implies that AdaMix can facilitate the model to delineate segmentation boundaries better. 
On the one hand, multi-task consistency-based approaches perform poorly in low data regions, i.e., 10% labeled data, indicating that the consistency regularization effects brought by auxiliary tasks, such as signed distance maps in SASSNet <cit.> and DTC <cit.>, or deep supervision in URPC <cit.>, are limited. 
On the other hand, due to uncontrollable perturbation strength, the models equipped with either random noise or CutMix show a performance gap compared to our AdaMix framework.
For instance, compared with the representative co-training-based model, MCNet+, our AdaMix-CT shows a performance gain of 1.48% and 1.32% in terms of DSC under 10% and 20% labeled data, respectively.
Moreover, our AdaMix frameworks significantly outperform SSL methods with fixed mix-up rules, i.e., UCMT <cit.> and ABD <cit.>. For example, compared to UCMT, AdaMix-CT obtains relative improvements of 2.62% in Dice similarity coefficient and 48.25% in average surface distance on the ACDC dataset with 10% labeled data.
These results demonstrate the superiority of Adaptive Mix for consistency regularization in semi-supervised medical image segmentation.





  §.§.§ Results on LA

We further evaluated the proposed method for left atrial segmentation in a 3D image segmentation scenario.
Table <ref> reports the results on the LA dataset with 10% (8 labeled samples) and 20% (16 labeled samples) labeled data.
In general, our AdaMix frameworks substantially outperform all previous methods in terms of the four metrics. 
Specifically, the AdaMix frameworks show competitive performance in surface-related metrics, indicating that AdaMix, through adaptively adjusting perturbation degrees, generates reasonable perturbed images, thereby facilitating the frameworks' advantages in 3D boundary segmentation.
For example, AdaMix-CT obtains the best ASD values of 1.57 and 1.44 under 10% and 20% labeled data, respectively.
These results prove the advantages of our method for semi-supervised volumetric medical image segmentation.




  §.§.§ Results on ISIC

Unlike previous datasets for cardiac structure segmentation from MRI images, the ISIC dataset involves skin lesion segmentation from dermoscopy images, which includes heterogeneous segmentation objects.
The results are reported in Table <ref>. 
It can be observed that 
in comparison with the methods utilizing CutMix <cit.> perturbation (e.g., FixMatch <cit.>, MT <cit.>, and CPS <cit.>) or UMix <cit.> perturbation (e.g., UCMT <cit.> and ABD <cit.>), our AdaMix frameworks demonstrate compelling performance.
For instance, compared with UCMT <cit.> and ABD <cit.>, AdaMix-CT shows performance gains of 1.64% and 1.26% in terms of DSC respectively under 5% labeled data. 
Remarkably, in light of boundary-sensitive metrics, i.e., 95HD and ASD, the proposed methods obtain the best scores due to effective training on the challenging target locations and boundaries in the perturbed images provided by AdaMix.
These results further demonstrate the superiority of our method for heterogeneous lesion segmentation.









Remark
One can observe from Table <ref>, Table <ref>, and Table <ref> that the results obtained by the three AdaMix frameworks are comparable, validating the flexibility of AdaMix to act as an effective plug-in-play module in SSL paradigms.
In contrast, the methods with different perturbation strategies show obvious performance discrepancies. For example, MT and AdaMix-MT exhibit a performance gap of 4.05% in terms of DSC on the LA dataset with 10% labeled data, while the margin between MT and CPS is only 0.08%.
The results suggest that perturbation strategies are more essential than learning paradigms for enhancing the models' performance in consistency regularization-based methods.  
This phenomenon can be attributed to the data-driven nature: DNNs are data-driven, making the quality of training data more crucial for accurately fitting the real data distribution in the underlying task, where the quality of training data can be considered as the degree to which the training data distribution represents the real data distribution.




 §.§ Ablation Study


In the following, we conducted extensive ablation experiments to investigate the contribution of each component of the proposed method in detail [Table <ref>]. Specifically, we took our AdaMix-ST as an example and examined the effects of AdaMix by considering three versions: 1) AdaMix-ST (V1): AdaMix-ST without the self-paced weight, 2) AdaMix-ST (V2): AdaMix-ST without the self-paced mask, and 3) AdaMix-ST (V3): the final method. Furthermore, we explored in-depth the influence of hyperparameters, i.e., the maximum number of mix-up patches K and the patch size S [Fig. <ref>].



  §.§.§ Improvement over the Supervised Baseline

Compared to the supervised baseline, our final method (i.e., AdaMix-ST (V3)) achieves considerable relative improvements of 5.97%, 9.99%, 31.80%, and 58.25%, in terms of DSC, Jaccard, 95HD, and ASD, respectively. 
These results demonstrate the effectiveness of our AdaMix frameworks for semi-supervised medical image segmentation. 



  §.§.§ Effectiveness of Each Component

Table <ref> shows a trend that the segmentation performance improves when we gradually incorporate the proposed components into our method.



  
Effectiveness of Self-Paced Mask
It can be observed that when a self-paced mask is introduced into the model to determine the mix-up rule based on the model's learning state, the results of AdaMix-ST (V1) are higher than those of SSL-ST, for example, with 2.08% and 1.06  improvement in terms of DSC and ASD, respectively.


  
Effectiveness of Self-Paced Weight
In the comparison between AdaMix-ST (V2) and SSL-ST, it can be seen that introducing the self-paced weight into our model to determine the number of mix-up patches results in a 2.25% improvement in DSC and a 1.17 improvement in terms of ASD.

We can conclude that our self-paced curriculum for determining both the mix-up rule and the number of patches is beneficial for semi-supervised segmentation. Building upon the self-paced curriculum, our final model, AdaMix-ST (V3), shows significant improvements over the semi-supervised baseline in global segmentation metrics (DSC and Jaccard) and boundary-sensitive metrics (95HD and ASD), achieving the best performance with 88.86% DSC, 81.03% Jaccard, 4.46 95HD, and 1.24 ASD.







 §.§ Analysis of Hyperparameters


We investigated the influence of the mix-up patch size S and the maximum number of mix-up patches K on AdaMix, using the ACDC dataset with 10% labeled data.
The results are shown in Fig. <ref>, where each subplot illustrates how the DSC changes with varying patch sizes when the maximum number of mix-up patches is fixed.
In general, AdaMix is not sensitive to changes in K, whereas variations in S cause slight performance fluctuations.
Specifically, increasing the patch size marginally improves performance, with the highest DSC score achieved when S = 32.
The main reasons behind this phenomenon are: 1) the proposed self-paced learning strategy can adaptively adjust the number of mix-up patches based on the model's learning state, where K only determines the maximum number of mix-up patches, and 2) patches that are too small or too large inevitably introduce noise to the images. Small patches fail to provide complete anatomical information, while large patches disrupt the anatomical structure of original images.
Moreover, it's surprising that the impact of the mix-up patch size K on the variation of segmentation performance is more trivial when the maximum number of mix-up patches K=32. We speculate that with a large number of mix-up patches, AdaMix can control a wider range of mix-up intensities. Consequently, it exhibits less sensitivity to hyperparameters, making the algorithm more robust.




 §.§ Analysis of Adaptive Mix Samples


AdaMix provides perturbed image examples with gradually increasing sample complexity, facilitating the effective consistency regularization on unlabeled data.  
As shown in Fig.<ref>, taking four cases from the ACDC dataset as an example, we illustrate the AdaMix procedure during training.
Overall, we can observe that high-confidence regions typically appear in the background, while low-confidence ones tend to appear at the boundaries of the segmentation targets. In medical image segmentation tasks, the background region contains simple patterns, while the object boundaries are considered complex ones <cit.>.
Deep neural networks learn simple patterns first <cit.>.
In the initial stage of training, considering the model's inferior capability, AdaMix provides perturbed samples with more high-confidence regions (i.e., more background regions), from which the model can generate relatively accurate predictions. 
As training progresses, the model's performance improves; AdaMix gradually incorporates low-confidence patches that generally appear at challenging target boundaries into the perturbed images, providing more complex samples for training; forcing the model to focus on these regions enhances its discriminative capability in the later training stages.
In light of the above, AdaMix can adaptively adjust the perturbation strength based on the model's learning state, thereby achieving more efficient consistency learning.








 §.§ Qualitative Results


As shown in Fig. <ref>, we visualize some segmentation examples on the ACDC, ISIC, and LA datasets. 
Specifically, the supervised baseline is prone to under-segment some lesion regions due to the limited number of labeled data. 
FixMatch <cit.>, MT <cit.>, and CPS <cit.> (all using CutMix <cit.> as the strong perturbation) incorrectly classify many boundary pixels, mainly due to noise introduced by random perturbations during training.
Similarly, UCMT <cit.> and ABD <cit.>, which use UMix <cit.> as the strong perturbation, also exhibit inferior performance on segmentation boundaries. These results can be attributed to the limited effectiveness of consistency regularization.
In contrast, our AdaMix-ST exhibits smoother and clearer boundaries compared to other methods. The finer segmentation results can be ascribed to the AdaMix algorithm which adaptively generates perturbed images in a self-paced learning manner.





§ CONCLUSION


This paper explores adaptive mix-up techniques for semi-supervised medical image segmentation. 
We propose a novel Adaptive Mix algorithm (AdaMix) for strong-weak pseudo supervision in semi-supervised learning. AdaMix features a self-paced curriculum that initially provides relatively simple mixed samples and then gradually increases the difficulty of the mixed images by adaptively controlling the mix-up strength based on the model's learning state. AdaMix can be seamlessly integrated as a plug-and-play module into self-training, mean-teacher, and co-training semi-supervised learning paradigms, achieving superior performance. Experiments on three public medical image datasets validate the effectiveness of the proposed method and demonstrate its superiority over state-of-the-art approaches.
Moreover, we found that replacing high-confidence regions with low-confidence ones, though seemingly counterintuitive, is effective for consistency regularization. Since low-confidence regions often appear at segmentation boundaries, forcing the model to learn from these areas helps enhance its discriminative capability. We also found that, in strong-weak pseudo supervision for semi-supervised learning, perturbation strategies are more essential than learning paradigms such as self-training, mean-teacher, and co-training.



§ CREDIT AUTHORSHIP CONTRIBUTION STATEMENT

Zhiqiang Shen: Writing – original draft, Methodology, Visualization, Validation, Data curation, Conceptualization.
Peng Cao: Writing – review & editing, Supervision, Project administration, Investigation, Funding acquisition, Formal analysis, Conceptualization.
Junming Su: Data curation, Visualization, Validation. 
Jinzhu Yang: Investigation, Funding acquisition.
Osmar Zaiane: Writing – review & editing, Formal analysis.



§ DECLARATION OF COMPETING INTEREST

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.



§ ACKNOWLEDGMENTS

This work was supported in part by the National Natural Science Foundation of China (Grant No. 62076059) and the Science and Technology Joint Project of Liaoning Province (Grant No. 2023JH2/101700367, ZX20240193).

elsarticle-harv




 Paper Title: VIPeR: Visual Incremental Place Recognition with Adaptive Mining and Lifelong Learning.





[
     R. Zamini[Corresponding author: rahelehzamini1361@yahoo.com] ^1,  S. Ghafouri^2  and F. Goodarzi^3

	^1  Department  of Mathematics, Faculty of Mathematical Sciences and  Computer, Kharazmi University, Thehran, Iran. 

	^2   Department of Mathematics, Faculty of Science, Arak University, Arak, 384817758, Iran.

^3   Department of Statistics, Faculty of Mathematical Sciences, University of Kashan, Kashan, Iran.

    
===================================================================================================================================================================================================================================================================================================================================================================================================================================

empty
empty



§ ABSTRACT 
Visual place recognition (VPR) is an essential component of many autonomous and augmented/virtual reality systems. It enables the systems to robustly localize themselves in large-scale environments. Existing VPR methods demonstrate attractive performance at the cost of heavy pre-training and limited generalizability. When deployed in unseen environments, these methods exhibit significant performance drops. Targeting this issue, we present VIPeR, a novel approach for visual incremental place recognition with the ability to adapt to new environments while retaining the performance of previous environments. We first introduce an adaptive mining strategy that balances the performance within a single environment and the generalizability across multiple environments. Then, to prevent catastrophic forgetting in lifelong learning, we draw inspiration from human memory systems and design a novel memory bank for our VIPeR. Our memory bank contains a sensory memory, a working memory and a long-term memory, with the first two focusing on the current environment and the last one for all previously visited environments. Additionally, we propose a probabilistic knowledge distillation to explicitly safeguard the previously learned knowledge. We evaluate our proposed VIPeR on three large-scale datasets, namely Oxford Robotcar, Nordland, and TartanAir. For comparison, we first set a baseline performance with naive finetuning. Then, several more recent lifelong learning methods are compared. Our VIPeR achieves better performance in almost all aspects with the biggest improvement of 13.65% in average performance. 







§ INTRODUCTION


Visual place recognition (VPR) aims to recognize a previously visited place by giving visual observations like images. It is a core component in simultaneous localization and mapping (SLAM) systems <cit.> to ensure the robust operation of autonomous agents and augmented/virtual reality (AR/VR) systems in large-scale environments. 
When solving the VPR task, most of the existing approaches <cit.> cast it as a retrieval task. The solution then typically involves a two-step process, with step one generating global descriptors from local features and step two matching the descriptors to the ones in a database of place-tagged descriptors.


Traditionally, VPR methods use hand-craft local features or semantic features and construct global descriptors with vector of locally aggregated descriptors (VLAD) <cit.>, graph-based descriptors <cit.>, etc. More recently, following the success of NetVLAD <cit.>, there have been notable advancements in end-to-end VPR methods in terms of accuracy <cit.> and robustness <cit.>. 
However, such a boost in performance comes at a cost. 
As the network architecture used in modern deep learning-based VPR methods getting much more complex, for example, KPConv <cit.> used in AEGIS-Net <cit.> and Transformer <cit.> used in R^2Former <cit.>, a laborious pre-training process that commonly takes days or even weeks becomes inevitable.
In addition, because the trained models have a strong independent and identical distribution (i.i.d.) assumption on the test data, the performance drops considerably when the test data is collected from an unseen environment that doesn't follow the i.i.d. assumption.







However, when deploying the VPR-equipped autonomous agents and AR/VR systems in real-world scenarios, it is difficult to collect sufficient data that covers all potential environments beforehand to pre-train the deep learning-based models. Such a gap between the advance in deep learning-based VPR methods and the practical needs of autonomous agents and AR/VR systems leads to the fact that recent practical-oriented systems still prefer the traditional VPR methods <cit.>. In this paper, we aim to close this gap by enabling the lifelong learning ability of the deep learning-based VPR method. In particular, we present VIPeR, a visual incremental place recognition method that can adapt to the newly observed environment during deployment while reserving its knowledge of previously visited environments. Fig. <ref> illustrates the core idea of the proposed VIPeR.



The idea of enabling lifelong learning for deep neural networks is not new. Also known as continual learning or incremental learning, it is designed to overcome the notorious catastrophic forgetting <cit.> when learning from sequential inputs. Despite that lifelong learning has been widely studied <cit.> and applied to many computer vision <cit.> or natural language processing <cit.> tasks, there is only a handful of works attempting to apply it to the place recognition task. 
AirLoop <cit.> and InCloud <cit.>, as the pioneers in lifelong place recognition, both tackle the catastrophic forgetting with a simple memory structure and employ knowledge distillation <cit.> to constrain the training. AirLoop further employs the memory aware synapses (MAS) <cit.> for regularization.



Although achieving promising performance, there is still plenty of room for improvement. As pointed out in CCL <cit.>, the naive triplet loss used in AirLoop <cit.> and InCloud <cit.> inevitably introduces bias when learning the global representation of various places. Such a phenomenon becomes even severe when hard-mining is applied to the training (see Table <ref>). In addition, the naive choice of a simple memory structure substantially hinders the power of the rehearsal strategy. Finally, the deterministic knowledge distillation used in AirLoop and InCloud is very sensitive to the outliers in the data, leading to potential performance degradation. 


Targeting these issues, we first introduce an adaptive mining scheme for triplet loss in metric learning. Our adaptive mining scheme can adaptively choose hard mining or soft mining, achieving a balanced performance between a single environment and multiple environments.
Focusing on lifelong learning, we then draw inspiration from human memory systems and design a novel memory bank to store memories of multiple environments. Our memory bank is made of three components. For the current environment, we have a sensory memory bank and a working memory bank. For previously visited environments, we have another long-term memory bank.
Finally, to make the model more robust against outliers, we extend the deterministic knowledge distillation used in AirLoop <cit.> and InCloud <cit.> to a probabilistic one.
We conduct extensive experiments on three large-scale datasets and our proposed VIPeR demonstrates superior performance when compared to other state-of-the-art (SoTA) lifelong place recognition methods.

In summary, our contributions are as follows: 

    
  * We introduce an adaptive mining scheme for metric learning to balance the performance within a single environment and 
    
    across multiple environments;
    
  * We design a novel, brain-inspired memory bank for rehearsal, alleviating the effects of catastrophic forgetting;
    
  * We propose a probabilistic knowledge distillation to regularize the training across different environments;
    
  * We conduct extensive experiments on three publicly available datasets and 
    demonstrate SoTA performance.
    






§ RELATED WORK




 §.§ Lifelong Learning



Lifelong learning targets the sequential learning setting and focuses on improving the adaptability and generalizability of the deep learning models. However, naive solutions like finetuning on the new tasks make the model quickly forget the knowledge about previous tasks, resulting in catastrophic forgetting <cit.>. Addressing this issue, most existing methods can be categorized into rehearsal-based approaches and regularization-based approaches <cit.>.

The core of the rehearsal-based approach is a memory buffer that either explicitly or implicitly stores samples from previous tasks. Then, when the model is trained on a new task, the samples stored in the buffer join the samples of the current task to refresh the model's memory about previous tasks. Exemplary works in this approach are iCaRL <cit.>, gradient episodic memory (GEM) <cit.>, and deep generative replay (DGR) <cit.>, which tackle the image classification task with iCaRL and GEM explicitly stores a small number of samples per class for rehearsal and DGR accompanies an additional generative model to enable implicit rehearsal.

Shifting the focus to the model itself, the regularization-based approach applies regularization on either the weight or the function. Regarding weight regularization, it commonly evaluates the importance of the model's parameters on previous tasks and mitigates the performance degradation by introducing a penalty for the loss. Within this scope, elastic weight consolidation (EWC)  <cit.> employs the Fisher information matrix to calculate the importance, while syntactic intelligence (SI)  <cit.>  and memory aware synapses (MAS) <cit.> both favor the online importance estimation. Then, in function regularization, various knowledge distillation approaches have been explored to distill the knowledge of old models to the new ones. An example work is learning without forgetting (LwF) <cit.>, which performs distillation by comparing the predictions from the output head of the old tasks to the ones of new tasks.







 §.§ Learning-based Place Recognition


NetVLAD <cit.>, as a milestone work in the learning-based VPR method, modified the VLAD <cit.> descriptor to make it differentiable and enabled end-to-end training of VPR methods. Demonstrating the representation power of the deep learning models, it inspires a plethora of works to promote the performance of deep learning-based VPR methods.

To improve the performance with image inputs, SPE-VLAD <cit.> explored the spatial pyramid structure of images to make the VLAD descriptor aware of structural information. 
Addressing the problems of appearance and viewpoint changes, Patch-NetVLAD <cit.> proposed to combine the advantages of local and global features by fusing multi-scale patch-level features that are computed from NetVLAD residuals.
Moving on to 3D inputs like point clouds, 
PointNetVLAD <cit.> is a straightforward extension of NetVLAD by replacing the 2D CNN with the PointNet <cit.> model to handle the input point clouds of fixed sizes. To empower the model to deal with inputs of random sizes, FusionVLAD <cit.> and AEGIS-Net <cit.> adopted more advanced 3D CNN with FusionVLAD focusing on cross-viewpoint VPR and AEGIS-Net concentrating on aggregating multi-level features. 

On the other hand, despite the fact that NetVLAD <cit.> is one of the most popular global descriptors, other global descriptors have also been explored. 
To start with, various forms of pooling have been employed for global descriptor generation due to their simplicity and generalizability. For example, APANet <cit.> favors sum pooling, Gordo et al. <cit.> chooses max pooling R-MAC <cit.>, MinkLoc3D <cit.> adopts generalized-mean pooling (GeM) <cit.>.
More recently, with Transformer <cit.> prevailing in a wide range of tasks, it has also been used to create global descriptors in VPR. For example, R^2Former <cit.> employs attention weights to guide the aggregations of local tokens.

Although tremendous advances have been witnessed in VPR research, most of the work tackles the task under the batch learning setting with the complete dataset available prior to the training. However, our paper investigates the VPR task under a sequential learning setting, which is addressed by a few works.

More related to our work, BioSLAM <cit.> tackles the VPR task from a lifelong learning perspective and chooses the generative replay from a rehearsal-based approach with a dual memory zone to store the previous observations. Nevertheless, AirLoop <cit.> combines a rehearsal-based approach with a regularization-based approach to mitigate the catastrophic forgetting issue with a lightweight model. Following this idea, InCloud <cit.> and CCL <cit.> extend to point cloud input with InCould introducing an angular-based knowledge distillation to relax the function regularization and CCL favoring contrast learning with InfoNCE <cit.> loss over metric learning with triplet loss <cit.>. 

Although we also build our work on AirLoop, our proposed VIPeR significantly differs from the above-mentioned methods with the adaptive mining for metric learning with triplet loss, a delicately designed three-zone memory bank for rehearsal, and probabilistic knowledge distillation (PKD) for regularization.






§ METHOD




As with many previous works on VPR, we cast the task of visual incremental place recognition as a retrieval problem. 
In particular, we define a sequence of environments ℰ = {E_0, E_1, ..., E_t, ...} with only one environment being available for training at a time. Furthermore, we define a set of images as observations for each environment E_t = {I_0, I_1, ..., I_i, ...}, which is also acquired in a sequential manner. 
Then, the VPR task is solved by incrementally constructing a database of place-tagged global descriptors 𝒢 = {g_0, ..., g_i, ... } with each one generated with g_i = f(I_i). 
f(·) here is a trainable model that consists of two sub-models, one extracts local features from input images and the other one aggregates the local features into global descriptors.
Whenever the query image I_q becomes available, the same model is used to generate a global descriptor from the query image d_q = f(I_q). Following this, by matching the query descriptor to the database descriptors, we can retrieve the top-K closest neighbors of the query image as the recognized places.
In the following, we present VIPeR to tackle visual incremental place recognition by combining metric learning with lifelong learning. An overview of VIPeR is shown in Fig. <ref>.



 §.§ Metric Learning



To train our model, we adopt metric learning with triplet loss. 
We first construct input triplet 𝒯 with an anchor image I^anc that represents a place in the environment, m positive images I^pos={I^pos_0, I^pos_1, ..., I^pos_m} that are taken from the same place, and n negative images I^neg={I^neg_0, I^neg_1, ..., I^neg_n} that are taken from different places.
Then, given the triplet 𝒯 = (I^anc, I^pos, I^neg), we drew inspiration from the naive triplet loss <cit.> and hard triplet loss <cit.>, and introduced adaptive triplet loss to train our model.





The naive triplet loss <cit.> is defined as 

    L_triplet=max( s^an_j-s^ap_i+δ ,0 ) ,

where s^an_j=sim(f(I^anc),f(I^neg_j)) is the cosine similarity between the global descriptors generated from the anchor image and a negative image sample, s^ap_i is the cosine similarity between the ones from the anchor and a positive image sample, δ is a hyper-parameter indicating the margin. When selecting the positive and negative image, the naive triplet loss opted for a random mining strategy with the sample indices i and j randomly selected. Despite that, the naive triplet loss exhibits better generalizability and is used in previous works like AirLoop <cit.> and InCloud <cit.>; it also makes the training process less effective, resulting in less discriminative global descriptors.  


The hard triplet loss <cit.>, to the contrary, goes to another extreme and favors the hard mining strategy. When choosing the sample images, the hard triplet loss focuses on the most difficult sample with the sample indices selected as i=argmin_i(s^ap_i) and j=argmax_j(s^an_j). Such a strategy largely increases the discrimination of the global descriptor and has been widely used by PointNetVLAD <cit.>, AEGIS-Net <cit.> for place recognition in a single large-scale environment. 

However, such a mining strategy inevitably introduces bias into the training, leading to performance degradation in lifelong learning settings. We later demonstrate this in Table <ref> in the ablation study.

To best join the advantages of random mining and hard mining, we propose a novel adaptive mining strategy and adjust the difficulty level of the training samples based on the fluctuations of the loss. We denote the triplet loss under the adaptive mining strategy as L_ada-triplet. Specifically, we keep tracking the changes in the loss during the training process. In the case when the loss increases in the current training step with the increment larger than a hyper-parameter T_d, we say the chosen training sample is too difficult, and we incline to a simpler sample in the following training steps. Contrarily, if the loss decreases in the current step with a decrement bigger than another hyper-parameter T_e, we determine the current training sample is too easy, and we lean towards more difficult ones. Under such a mining design, we achieve a balance between the discrimination requirements for place recognition and the generalizability requirements for handling the sequential input.




 §.§ Brain-inspired Memory Bank



To alleviate the notorious catastrophic forgetting in lifelong learning, we first investigate the rehearsal-based approach to refresh the model's memory about previous environments. Specifically, we present a brain-inspired memory bank ℬ=(M, R) with a fixed-size image storage space M and a corresponding adjacency matrix R.
Each element in the adjacency matrix R_i,j represents the relative location between place i and place j, which is then used to determine whether these two places belong to a positive or a negative pair.

Regarding the fixed-size image storage space M, we mimic the human memory system and design the architecture as a composition of a per-environment sensory memory M^sn, a per-environment working memory M^wk, and an environment-agnostic long-term memory M^lt.
Our brain-inspired memory bank is capable of improving the model's adaptability in a new environment while preserving its stability over all previous environments. We illustrate the update process of the proposed brain-inspired memory bank in Fig. <ref>. 

For any unseen environment t, the sensory memory M^sn_t, which is a first-in-first-out queue of size l^sn, is always first activated and used to store the most recently visited places. With such a design, we ensure the model stays sensitive to the latest changes in the scene. 
Whenever the sensory memory is full, the working memory M^wk_t, as a list of size l^wk, is activated to store the places coming out of the sensory memory. When the working memory is also full, we decide on whether the new place should be stored with a probability p=l^wk / num, where num is the total number of images that have passed in the current environment. This makes the working memory a perfect complement to the sensory memory since the probability of updating the working memory keeps decreasing as more images are processed, allowing the old observations to be preserved.

The long-term memory M^lt, also as a list of size l^lt, stores the samples of places from all previously visited environments to ensure its environment-agnostic property. 
When entering a new environment t, the long-term memory M_t^lt is updated following

    M_t^lt=ω M_t-1^wk+( 1-ω) M_t-1^lt ,

where M^wk_t-1 and M^lt_t-1 are the working memory and the long-term of environment t-1, ω is a hyper-parameter to control the update ratio.









  




 §.§ Relational Memory Aware Synapses

Additionally, we explore a weight regularization approach to prevent undesired weight updates when the model is adapting to the new environment.
Particularly, we follow the relational memory aware synapses (RMAS) proposed in AirLoop <cit.> to estimate the importance weight Ω^RMAS_t of each parameter θ in the environment t. Then, the changes in the parameters are penalized with a regularization loss L_RMAS. Note that for simplicity, we omit the subscript t to index the environment in the loss function. We hereby briefly review the formulation of RMAS. For more details, we refer the readers to the original AirLoop <cit.> paper. 

For environment t, the weight is approximated as

    Ω^RMAS_t≈1/N_t∑_k=1^N_t( ∂‖S̃_k, t‖_F/∂θ) ^2,

where N_t is the number of images that have been processed, ‖·‖_F is the Frobenius norm,  S̃_k, t∈ℝ^3×3 is the Gram matrix at the k-th training step. Each element in the Gram matrix is computed using the triplets we selected with adaptive mining in Section <ref>, following S̃_k, t|(0,0) = s^aa, S̃_k, t|(0,1) = s^ap_i, S̃_k, t|(0,2) = s^an_j, etc.. 


Then, the RMAS loss is computed as 


    L_RMAS=∑_θ∈Θ^n Ω^RMAS_t-1( θ_t - θ_t-1) ^2 ,

where Θ is the set of all parameters, θ_t and θ_t-1 indicate the updated parameter in the current environment t and the frozen parameter after training on environments 1, 2, ..., t-1.






 §.§ Probabilistic Knowledge Distillation

Moreover, we delve into the function regularization approach with knowledge distillation to further enhance the model's adaptability to unseen environments during lifelong learning. 
Although previous works like AirLoop <cit.> and InCloud <cit.> have also explored knowledge distillation, we believe that their performance is limited due to two reasons. First of all, when selecting memory samples for distillation, these works use images from the current environment. We claim that such a choice is counter-intuitive as the previous model hasn't seen the current environment, and its performance cannot be guaranteed. Secondly, these works opt for deterministic distillation with angular metrics, which makes the model more focused on local structures of samples.  

To select more effective samples, we propose to only use the samples from the long-term memory M^lt of our brain-inspired memory bank to perform knowledge distillation.
Besides, we present PKD to make the model pay more attention to the global distribution of the samples. To estimate the distribution of the samples in the learned feature space, we first construct the matrices H_t and H_t-1 using the model trained in the current environment t and the frozen model that has been trained on environment 1, 2, ..., t-1 respectively. Specifically, elements in the matrix are computed as



    H_Ξ|i,j=g_i, Ξ·g_j, Ξ^T/√(d)   for   Ξ∈ [t-1, t] ,

where d is the dimension of the global descriptor g. Although similar ideas can be seen in CCL <cit.>, we notice that their design with the distillation temperature hyper-parameter easily leads to a vanishing gradient during the training process. We are inspired by Transformer <cit.> and make a simple yet effective modification by replacing the distillation temperature hyper-parameter with a model-specific value √(d) to prevent the gradients vanishing issues.

Once the H matrices are computed, we can estimate the distribution with the softmax (SM) function and calculate the PKD loss with Kullback-Leibler (KL) divergence as 


    L_PKD=∑_i,jKL{SM( H_t-1|i,j) log(SM( H_t|i,j) ) } .




 §.§ Final Loss

Finally, we combine the above-mentioned loss functions into the final loss function of the following form



    L=L_ada-triplet + λ_1 L_RMAS + λ_2 L_PKD,

where λ_1 and λ_2 are hyper-parameters used to adjust the contributions of the RMAS and PKD loss functions.




§ EXPERIMENT



 §.§ Implementation Details

When choosing the VPR method, we first experiment with the same VPR method used in AirLoop <cit.>, which consists of a pre-trained VGG-19 <cit.> for local feature extraction and GeM <cit.> for global descriptor aggregation.
The hyper-parameter in GeM also follows AirLoop as p=3, and the dimension of its global descriptor is set to d=1024. Additionally, to get more discriminative global descriptors, we extend the VPR method by replacing the GeM layer with a NetVLAD layer <cit.>. We especially follow many other VPR methods like <cit.> and set the number of clusters to K=64. However, we set the dimension of global descriptors generated by NetVLAD to be the same as that of GeM descriptors. 


When forming the triplet tuple, we empirically set the number of positive and negative samples to be m=1 and n=5. And we set the margin in Eq. <ref> to be δ=1.
Regarding adaptive mining, we prevent the model from always learning with easy samples by explicitly setting the threshold for using easier samples to be higher than the one for using more difficult ones.
Especially, we choose T_d=0.02 and T_e=0.01.


As for the brain-inspired memory bank, 
we ensure a fair comparison by setting the total size of our memory bank to be the same as the queue's length in AirLoop <cit.>. Particularly, we empirically divide the total size into l^sn=500 for sensory memory, l^wk=400 for working memory, and l^lt=100 for long-term memory. In addition, we set the hyper-parameter in Eq. <ref> to be ω=0.5. 

Finally, for Eq. <ref>, we choose equal weights with hyper-parameters λ_1=1 and λ_2=1.
We train the VPR methods using the stochastic gradient descent (SGD) optimizer with a learning rate of 0.002 and a momentum of 0.9.









 §.§ Datasets and Evaluation Metrics


Datasets: We evaluate our proposed VIPeR on three large-scale, publicly available datasets, namely Oxford RobotCar <cit.>, Nordland <cit.>, and TartanAir <cit.>.

Oxford RobotCar <cit.> is a real-world autonomous driving dataset with data collected under various weather conditions across multiple urban environments in Oxford. 

For evaluation, we follow AirLoop <cit.> and select three environments under the tag “sun”, “overcast”, and “night”. For each environment, the same two sequences are selected for training and testing. 
We define two images belonging to the same place if the distance between them is less than 10m and the yaw difference is less than 15^∘.

Nordland <cit.> is also a real-world dataset with across-season data collected from a railway journey through the Nordland region of Norway. 

As AirLoop <cit.>, we also use the recommended train-test split and determine whether two images are taken from the same place if they are separated by three images or less.

TartanAir <cit.> is a photo-realistic synthetic dataset. It covers both indoor and outdoor environments and provides data collected under diverse lighting conditions, weather conditions, and times of the day.
Also following AirLoop <cit.>, we select 5 environments for evaluation with the same surface intersection-over-union (sIoU) used as the criteria to decide whether two images are from the same place. Specifically, we set sIoU<0.1 for negative, sIoU>0.7, and sIoU>0.5 for positive in training and test, respectively.





Evaluation metrics: In line with AirLoop <cit.>, we use the recall rate at 100% precision as the core metric to evaluate the model's performance on single environments. 
To better assess the performance in lifelong learning, we evaluate the model trained after each environment on every environment in the environment sequence ℰ of length T, even the unseen ones. We summarize the results in an evaluation matrix P ∈ℝ^T× T with each of its element P_i,j being the recall rate at 100% precision when evaluating the model trained after environment i in environment j.

Given the evaluation matrix, we further compute the average performance (AP), backward transfer (BWT), and forward transfer (FWT), which are introduced in <cit.>, to evaluate the overall performance, how does learning in new environments affect the previous knowledge, and how well does the model generalize to unseen environments. We refer readers to <cit.> for the detailed formulas for computation.













 §.§ Evaluation and Discussions

For comparison, we first set a baseline performance with naive finetuning on both of the VPR methods we experimented with. In addition, we also explore the performance of two generic weight regularization methods in lifelong learning, EWC <cit.> and SI <cit.>. For more recent methods, we compare our proposed VIPeR to AirLoop <cit.> and InCloud <cit.>, which are specifically designed for visual incremental place recognition. However, because AirLoop only provided the performance with the combination of VGG-19 <cit.> and GeM <cit.>, we re-train AirLoop with their official implementation to get its performance with the combination of VGG-19 and NetVLAD <cit.>. Similarly, since InCloud was only tested with LiDAR point clouds, we replaced its VPR methods with the same one that our VIPeR experimented with and re-trained the InCloud model with their official implementation.



We present the qualitative results in Fig. <ref> with VIPeR, AirLoop <cit.>, and InCloud <cit.> trained for all available environments and evaluated on the very first one to demonstrate the model's resilience to catastrophic forgetting. 
Then, we also present quantitative results in all three environments in Table <ref>.  
It can be seen that naive finetuning and generic weight regularization method EWC <cit.> and SI <cit.> exhibit inferior performance, especially in AP and BWT, indicating their inadequate abilities when fighting against catastrophic forgetting in visual incremental place recognition.
Furthermore, despite that both AirLoop <cit.> and InCloud <cit.> achieved improved performance over baseline methods by utilizing a combination of several lifelong learning methods to mitigate the catastrophic forgetting, they suffer from a notable performance drop when changing from GeM <cit.> to more discriminative NetVLAD <cit.>.

The results of our proposed VIPeR exhibit superior performance over other methods in almost all metrics.
In particular, our VIPeR prevails in terms of AP and FWT across all three datasets, which implies the model's strong performance and good generalizability towards unseen environments.
In terms of BWT, which measures the forgetness of the model, our VIPeR didn't to achieve the best performance. However, we argue this amount forgetting is acceptable as even after forgetting some knowledge of the previous visited place, our VIPeR still exhibits better recognition performance when compared to AirLoop and InCloud.
Additionally, when changing the global descriptor aggregator from GeM to NetVLAD, our VIPeR, unlike AirLoop <cit.> or InCloud <cit.>, achieves even better performance, which further demonstrates the effectiveness of presented solutions.













 §.§ Ablation Study

We further investigate the contribution of the adaptive mining, brain-inspired memory bank, and PKD loss by replacing the corresponding modules in the base model, i.e. AirLoop <cit.>, one at a time. 

Adaptive Mining: To start with, we replace the random mining with hard mining, which is commonly used in VPR tasks, and the proposed adaptive mining. We conduct experiments in all three datasets as above and present the results in Fig. <ref> and Table <ref>. 
The adaptive mining strategy used in our proposed VIPeR exhibits favorable performances in TartanAir <cit.> and Oxford RobotCar <cit.>, outperforming random mining used in the base model and hard mining used in VPR methods with batched inputs.
Although our adaptive mining does not perform as good as hard mining in Nordland <cit.> when we increasing the number of negative samples, we believe it is mainly caused by the repetitive scenes between different environments in Nordland. We argue that hard mining introduces serious bias into the learning process, which is demonstrated by the drastic performance drop in TartanAir and Oxford RobotCar.









Brain-inspired Memory Bank: Then, we replace the naive queue used in the base model with the brain-inspired memory bank in the proposed VIPeR and present the quantitative results on Oxford RobotCar <cit.> dataset in Table  <ref>. It is notable that the brain-inspired memory bank surpasses the naive queue with a large margin across all evaluation metrics. The results demonstrate the superior performance of the brain-inspired memory bank in lifelong learning with good generalizability and resilience to catastrophic forgetting.




Probabilistic Knowledge Distillation: Finally, replace the relational knowledge distillation in the base model with the PKD used in the proposed VIPeR. In addition, we also compare with performance of angular-based knowledge distillation proposed in InCloud <cit.>. 
As shown in Table <ref>, it is evident that our probabilistic distillation largely exceeds the performance of relational knowledge distillation and angular-based knowledge distillation, let alone the naive finetuning strategy.
The extraordinary results validate that our distillation approach is capable of grasping the underlying data distribution, which is a big advantage in dealing with catastrophic forgetting and the adaptability when training in unseen environments.









§ CONCLUSIONS

In this work, we propose VIPeR, a visual incremental place recognition method. We investigate
triplet mining strategies suitable for lifelong learning and find a balance between per-environment and cross-environment performance with the proposed adaptive mining. In addition, we mitigate the catastrophic forgetting issues by combining a rehearsal-based approach with a regularization-based approach. In particular, we design a novel brain-inspired memory bank to mimic the human memory system and present a probabilistic knowledge distillation.
We conduct extensive experiments on large-scale datasets, the results of which demonstrate the effectiveness of the proposed VIPeR.
As for future work, we believe there are many possible directions, like extending the VIPeR model to handle other forms of single-modality input or cross-modality input.  
















ieeetr







 Paper Title: Towards Reliable Advertising Image Generation Using Human Feedback.





 










Z. Du et al.




School of Artificial Intelligence and Automation,
 Huazhong University of Science and Technology, Wuhan, China 

dzb99@hust.edu.cn Retail Platform Operation and Marketing Center, JD, Beijing, China 

 {fengwei25, wanghaohan1, liyaoyu1, wangjingsen, lijian21, zhangzheng11, lvjingjing1, zhuxin3, jinjunsheng1, shenjunjie, linzhangang, shaojingping}@jd.com

Towards Reliable Advertising Image Generation Using Human Feedback
    Zhenbang Du1, 2Equal contributionWork done while interning at JD.com0000-0002-1386-8381 Wei Feng2^* Haohan Wang2 Yaoyu Li2Jingsen Wang2 Jian Li2 Zheng Zhang2 Jingjing Lv2 Xin Zhu2 Junsheng Jin2 Junjie Shen2 Zhangang Lin2 Jingping Shao2
    August, 2024
===============================================================================================================================================================================================================================================




§ ABSTRACT 
    In the e-commerce realm, compelling advertising images are pivotal for attracting customer attention. While generative models automate image generation, they often produce substandard images that may mislead customers and require significant labor costs to inspect. This paper delves into increasing the rate of available generated images. We first introduce a multi-modal Reliable Feedback Network (RFNet) to automatically inspect the generated images. Combining the RFNet into a recurrent process, Recurrent Generation, results in a higher number of available advertising images. To further enhance production efficiency, we fine-tune diffusion models with an innovative Consistent Condition regularization utilizing the feedback from RFNet (RFFT). This results in a remarkable increase in the available rate of generated images, reducing the number of attempts in Recurrent Generation, and providing a highly efficient production process without sacrificing visual appeal. We also construct a Reliable Feedback 1 Million (RF1M) dataset which comprises over one million generated advertising images annotated by human, which helps to train RFNet to accurately assess the availability of generated images and faithfully reflect the human feedback. Generally speaking, our approach offers a reliable solution for advertising image generation. Our dataset and code are available at <https://github.com/ZhenbangDu/Reliable_AD>. 
  





§ INTRODUCTION



An attractive advertising image is essential for e-commerce success, as it can lead to a higher click-through rate (CTR) <cit.>.  Manual creation requires significant labor costs, therefore the demand for automatic advertising image production is on the rise.  However, previous approaches <cit.> often result in visual mismatches between the product and background. Advanced diffusion models <cit.> offer a remedy. A combination with ControlNet <cit.> has shown promise in producing harmonious backgrounds for products while keeping the details of products identical for advertising use <cit.>.

Despite the potential of generative models to create appealing backgrounds, we have observed a frequent production of substandard advertising images as depicted in <ref>, which encompass various cases such as space and size mismatches, indistinctiveness, and shape hallucination. These flawed images can lead to customer misunderstandings about products and bring a subpar shopping experience, therefore considerable labor is necessitated to inspect the generated images. Such drawbacks limit the broader application of generative models in advertising image production. So the core problem is a low rate of available images. It poses a new challenge to us how to establish a reliable advertising image generation pipeline capable of producing images with a high available rate. 



A natural solution is to generate images repeatedly until an available image is obtained (Recurrent Generation) due to the randomness in generation. To substitute labor inspection in this repetitive process, a novel Reliable Feedback Network (RFNet) is introduced to act as a human inspector, assessing the availability of generated advertising images. As simply depending on a single generated image, the model can not effectively obtain the knowledge pivotal to the precise inspection, , what the product is and how the product appears in the background. So RFNet integrates multiple auxiliary modalities to provide critical information contributing to the judgment of different unavailable cases. Meanwhile, we construct a large-scale dataset called Reliable Feedback 1 Million dataset (RF1M), which includes over one million elaborate generated advertising images with rich human annotations, playing a crucial role in training RFNet to mirror human feedback accurately.

While Recurrent Generation greatly increases the number of available images, multiple attempts will dramatically prolong the generation process owing to the inherent poor ability of generated models. Using human feedback to enhance the capability of diffusion models <cit.> provides a viable option, which has successfully improved the visual quality of generated images <cit.>. However, the visual quality and availability of the generated images show a trade-off relationship, , products with repetitive and simple backgrounds gain a high available rate yet low aesthetics. To tackle this, we propose a novel loss term, Consistent Condition (CC) regularization L_CC to counteract uniformity and degradation of the generated backgrounds, circumventing the adversarial nature of conventional Kullback-Leibler (KL) regularization <cit.>. Utilizing this regularization term, the feedback on generated images' deviation from the available type assessed by RFNet is directly back-propagated to fine-tune the diffusion model (RFFT). Our approach significantly enhances the available rate of generated advertising images without altering their aesthetics, offering a comprehensive solution to the challenge of reliable advertising image generation.


Our main contributions include:

    
  * An advertising image generation solution Recurrent Generation alongside the novel multi-modal model, RFNet, which simulates human feedback and effectively utilizes various modalities to help attribute fine-grained issue types.
    
  * A straightforward and effective approach, RFFT, to refine the diffusion model using human feedback, along with an innovative Consistent Condition regularization to prevent collapse.
     
  * A large-scale multi-modal dataset, RF1M, comprising over one million generated advertising images with rich annotations.




§ RELATED WORK



In this section, we review prior works on advertising image generation and the use of human feedback to refine diffusion models.



 §.§ Advertising Image Generation


Automatic advertising image generation offers significant convenience for e-commerce. Current approaches generally fall into two folds, template-based and generation-based. Template-based approaches <cit.> typically utilized predefined templates to assemble various elements into images. However, such approaches failed to achieve aesthetic harmony between the product and its background, resulting in a discernible disconnection. Additionally, designing a variety of templates was costly. To counter this, generation-based approaches <cit.> employed GANs <cit.> or advanced text-to-image diffusion models <cit.> to produce backgrounds that coordinate with the appearance of products, , Wang <cit.> made use of the inpainting technique <cit.> with product masks to produce personalized backgrounds. Although elaborate and coordinated advertising images were obtained by these approaches, the bad cases that happened in the generation process could result in confusion and complaints of customers. We tackle this issue with a Recurrent Generation strategy, complemented by the accurate inspection capability of RFNet. Additionally, our RFFT enhances the reliability of the diffusion model and further promotes the efficiency of the production process.



 §.§ Refining Diffusion Models with Human Feedback


Reinforcement Learning from Human Feedback (RLHF) employs human-derived feedback to fine-tune models, aiming for outcomes that human prefers <cit.>. In aligning diffusion models with human preferences, RLHF plays a crucial role <cit.>. Works such as DDPO <cit.> and DPOK <cit.> treated the denoising process as a multi-step Markov decision process. They used a Policy Gradient <cit.> approach to update model parameters based on feedback from pre-trained reward models. Similarly, D3PO <cit.> and Diffusion-DPO <cit.> enhanced diffusion models using human comparison data, avoiding the need for reward model training. Contrary to the reinforcement learning strategy, ReFL <cit.> and DRaFT <cit.> directly fine-tuned diffusion models using the gradient of differentiable rewards in an efficient end-to-end manner <cit.>. Our work pioneers in using human feedback to establish a reliable advertising image generation solution. Additionally, our approach addresses the collapse during fine-tuning, resulting in a high available rate of generated images without compromising their appearance.



§ DATASET

The Reliable Feedback 1 Million (RF1M) dataset, constructed through meticulous human feedback, serves as a pivotal resource for inspecting and improving the generation of advertising images. Compared to prevailing large-scale image generation dataset Laion-5B <cit.>, DiffusionDB <cit.> and Pick-a-Pic <cit.>, RF1M is specifically designed for the advertising domain, addressing the acute need for expansive data resources in this field. Here’s an in-depth look at its composition, annotation, and potential impact on the community.



Composition. The dataset is generated using a collection of extensive products from JD.com. It encompasses 1,058,230 samples, each consisting of a variety of components aimed at providing a comprehensive understanding of advertising image generation:

    
  * The generated advertising image with corresponding transparent background product image, and carefully designed prompts by professional designers.
    
  * Depth and salience images created by the dense prediction transformer <cit.> and U2-Net <cit.> trained on e-commerce data, along with product caption, which assists in inspecting the availability of generated advertising image.
    
  * Human-annotated label indicating the availability of image for advertising use.


These elements collectively offer a rich foundation for analyzing the generated advertising images. Some examples are exhibited in Fig. <ref>.
 
Annotation. The annotators involved are well-versed in advertising and possess a deep understanding of the standards for advertising images. Within the dataset, samples have been further classified into five fine-grained categories as illustrated in Fig. <ref>:

    
  * Available. Images deemed suitable for advertising purposes.
    
  * Space Mismatch. Images where the product and background have inappropriate spatial relations, such as a part of the product is floating.
    
  * Size Mismatch. Discrepancies between the product size and its background, , a massage chair appears smaller than a cabinet.
    
  * Indistinctiveness. Images where the product fails to stand out due to background complexity or color similarities.
    
  * Shape Hallucination. Backgrounds that erroneously extend the product shape, adding elements like pedestals or legs.



Potential Impact. RF1M, with its multi-modal design and comprehensive features, is the cornerstone of the RFNet training and RFFT, and would further significantly impact the realm of e-commerce advertising and beyond, featuring three highlights: 


    
  * Large scale. With its vast array of product categories and image types, RF1M surpasses previous advertising datasets BG60k <cit.>, PPG30k <cit.> and human feedback dataset ImageRewardDB <cit.>, RichHF-18K <cit.>, while has comparable size to Pick-a-Pic <cit.>, providing a robust base for RFNet to mirror human feedback accurately across the diverse advertising image generation tasks. 
    
    
  * Scalability. The multi-modal nature of RF1M provides sufficient information to RFNet to make judgments precisely. Beyond generating reliable advertising images, it also supports tasks such as advanced image understanding and image matting. This flexibility ensures the dataset meets the changing demands and can be applied in numerous areas within and beyond advertising.

    
  * Visual appeal. The prompts and generation models are carefully designed for the products' characteristics, so the images have satisfying aesthetics and can attract customers' attention. We conducted a one-week online A/B test done in JD.com, resulting in a 2.2% increase in CTR from over 60 million exposures, underscoring the high quality of these images, which accurately captured user preferences.







§ METHODOLOGY




 §.§ Preliminaries

Our approach for generating advertising images is depicted in Fig. <ref>. We start with a text prompt describing the desired background and a product image I_o with a transparent background. The prompt is input into Stable Diffusion <cit.>, and I_o is pre-processed with canny edge detection before being fed into the ControlNet <cit.>. We adopt DDIM <cit.> as our denoising schedule, the latent representation x at step t is calculated as following:

    x_t=√(α̅_t)x_t+1-√(1-α̅_t+1)ϵ_θ(x_t+1, t+1)/√(α̅_t+1)+√(1-α̅_t)ϵ_θ(x_t+1, t+1),

where ϵ_θ represents the model <cit.> that predicts added noise, and {α̅} is a set of coefficients controlling the forward noise-adding process. To preserve the product's integrity and ensure a cohesive background, we employ an inpainting technique <cit.>. The latent representation x_t is processed by:

    x_t= (I-M)⊗ x_t + M⊗ x_o,

where x_o is the latent of I_o, M is product mask, and ⊗ denotes the element-wise multiplication. After the x_0 is obtained, this latent is converted to the generated image I_g.




 §.§ Recurrent Generation with RFNet


Due to the inherent randomness, repeated generation can significantly expand the number of available images. To automate the inspection process and eliminate human participation in this process, we introduce a multi-modal model, RFNet, to determine whether the generated image is available precisely, as illustrated in Fig. <ref>. In addition to I_o and I_g, RFNet combines information from auxiliary modalities:

    
  ∙ The depth image I_d of I_g produced by a depth estimation model, highlights the product's position relative to the background;
    
  ∙ The salience image I_s of I_g, created by a salience detection model, outlines the product;
    
  ∙ The product caption Cap which provides insight into the product's attributes.


I_o, I_g, I_d and I_s are fed into an image encoder to acquire respective image embeddings {e_o, e_g, e_d, e_s}. Concurrently, the Cap is input into BERT <cit.> to obtain the text embedding e_c,  aiding in recognizing the attributes of the product.



Since product captions often contain excessive information, , brands, we focus on distilling the vision-related attributes from the caption. So N_1 Feature Filter Module (FFM)s are first employed, each consisting of a cross-attention layer and several convolution layers. The output of FFM is formulated as: 

    e_f = Conv(Conv(CrossAttention(e_o, e_c)) ⊗Conv(e_o))+ e_o,

where e_o serves as Query with e_c acting as both Key and Value in cross-attention layer, Conv() denotes convolution layer with 1×1 kernels, and ⊗ signifies element-wise multiplication. This process ensures that critical information from the caption is effectively integrated with the image embeddings, augmenting the model's understanding of the product.

With the fused embedding e_f, different features are further integrated through N_2 self-attention layers <cit.>,

    f = SelfAttention(Concat(e_f, e_g, e_d, e_s)),

where Concat() stands for concatenation. These stacked layers capture the critical features across the embeddings. Finally, a fully-connected classifier determines each case's probability of the generated image. 

Trained on large-scale RF1M, RFNet assesses the availability of generated advertising images accurately by considering a comprehensive set of visual and textual features, and providing nuanced feedback. This capability, combined with the Recurrent Generation strategy (illustrated in <ref>), significantly increases the number of available generated images for advertising use in an automatic manner.



 §.§ RFFT with Consistent Condition regularization


Although Recurrent Generation could produce more available images in total, the inherited poor ability of the generative model results in a prolonged and inefficient production process, posing a great challenge to the application. Our end-to-end generation-inspection pipeline allows feedback gradients from RFNet to directly fine-tune the diffusion model, enhancing its capability. Specifically, our proposed RFFT selects a random step t among last 10 steps during the 40-step denoising process to generate the x̂_0^t=x_t+1-√(1-α̅_t+1)ϵ_θ(x_t+1, t+1)/√(α̅_t+1) <cit.>. The resulting x̂_0^t is post-processed to Î_g^t then inspected by RFNet to determine its availability, with feedback calculated as follows:

    F_AC = -1/N∑_i=1^Ny_dlog(ô_i),

where y_d is the one-hot vector representing the desired “Available” category, the vector ô_i holds the probability for each generated image case and N is the total number of samples. The gradient ∇ F_AC is then back-propagated to steer the model towards producing images with a higher probability of being available.


Nevertheless, the training goal of making the model reliable runs counter to the aesthetics of the images generated by the model, , repetitive and simple backgrounds can perfectly circumvent the bad cases we mentioned before. So as illustrated in Fig. <ref>.a, as training progresses, the model achieves an extremely high available rate, yet yields homogeneous and aesthetically collapsed outputs. So a training approach that keeps the aesthetics of the image stable while increasing its availability rate is what we need. A prevalent solution involves the Kullback-Leibler (KL) regularization <cit.>, a loss term that ensures the modified model does not diverge significantly from the desired distribution, thus maintaining diversity and preventing the convergence to sub-optimal, repetitive results, and this loss term can be formulated as:

    L_KL = KL(p_θ(x̂_0^t|x_t,z,c)‖(p_ref(x̂_0^t|x_t,z,c)),

where c and z are image and text control conditions, p_θ and p_ref represent the distributions of current and reference models. However, as the feedback gradient endeavors to steer the image generation towards a higher available rate, the KL regularization strives to maintain the generated image unchangeable. This opposition mirrors the principles of adversarial training <cit.> and poses a challenge to achieve a win-win solution <cit.>.




Instead of focusing on unchanged images, we aim to maintain visual quality. For text-to-image generation, the visual output is closely linked to the input text condition z <cit.>. In a classifier-free manner, we derive text guidance from the model's implicit classifier <cit.> by

    ∇_x_tlogp_θ^t(z|x_t, c) ≈ - 1/√(1-α̅_t)(ϵ_θ(x_t, z, c) - ϵ_θ(x_t, c)),

which indicates the direction where the text condition influences image generation. To ensure improvements in image availability do not compromise the core conditions, we introduce a Consistent Condition (CC) regularization term L_CC, as follows:

    L_CC = ‖∇_x_tlogp_θ^t(z|x_t, c) - ∇_x_tlogp_ref^t(z|x_t, c)‖_2.


Fig. <ref>.b illustrates the advantage of L_CC over L_KL. While L_KL acts to limit updates from ∇ F_AC potentially leading to rigidity, L_CC offers a win-win approach. It maintains the direction of the condition, allowing for the model to be fine-tuned towards generating more available images. Thus the final feedback to fine-tune the diffusion model in RFFT is:

    F_total = F_AC+β L_CC,

where β is a hyper-parameter.





§ EXPERIMENTS




 §.§ Implementation Details

For RFNet, we employ a ResNet50 <cit.>, pre-trained on ImageNet <cit.>, as the image encoder. RoBERTa <cit.>, fine-tuned on Chinese product descriptions, extracts text embeddings from product captions. We resize all images to 384×384 before encoding. The FFM and Self-Attention contain blocks of width 384 with 8 attention heads, and we set N_1 and N_2 to 1 and 3, respectively. The training spans 10 epochs, starting with a learning rate of 1e-4, which is reduced by a factor of 10 at epoch 5.

For diffusion model fine-tuning, we utilize 8 NVIDIA A100 GPUs, a local batch size of 4, and 4 gradient accumulation steps. We opt for AdamW <cit.> with a learning rate of 1e-5. Unless specified otherwise, MajicmixRealistic_v7 (Maji_v7)[<https://civitai.com/models/43331/majicmix-realistic>] serves as the diffusion model, complemented by ControlNet V1.1 <cit.>[<https://github.com/lllyasviel/ControlNet>]. During fine-tuning, only ControlNet is trained, while we freeze the remaining parameters. A 40-step DDIM with the last 10 steps chosen for fine-tuning is used <cit.>.






 §.§ Advertising Image Inspection Performance

SOTA Approaches. To validate the superiority of our proposed RFNet, we implement prevailing models ResNet50 <cit.>, ResNeXt50 <cit.>, HRNet <cit.> and Vision Transformer (ViT) <cit.> to inspect the generated images.

Evaluation Metrics. Precision, recall, F1 score, and average precision (AP) are used to evaluate the prediction results of different models. We conduct the test on 1,000 images. It is noteworthy that since we are only concerned about the availability of the generated images in Recurrent Generation, we focus on whether the model is able to accurately identify the available images.

Results. Table <ref> shows that RFNet outperforms across all metrics, highlighting the benefits of integrating multi-modal information and its effective structure. We conducted extensive experiments to evaluate the impact of various components within RFNet, presented in Table <ref>. Our experiments demonstrate the significant impact of each component within RFNet on the final AP, especially the crucial role of product images, whose ablation leads to a notable 6.41 drop in AP. We further trained the model using coarse-grained labels (Available/Unavailable), the decrease showcased that a fine-grained label helped to attribute the issues precisely, leading to improved performance. 








 §.§ Advertising Image Reliability Performance



SOTA Approaches. 
To evaluate the effectiveness of refining diffusion models, our approach is compared with Ori (using the original model), Prompt Engineering (PromptEng) (using modified prompts)[Add “irregular shape, extended shape, floating, table legs, pedestal, improper position, improper size, indistinct background” to the negative prompt <cit.>.], and SOTA RLHF approaches PPO <cit.>, DPO <cit.>, and ReFL <cit.>.

Evaluation Metrics. 
For availability evaluation, we conduct the test on 1,000 products. The available rate is defined as the ratio of images deemed available by our RFNet to the total number of images inspected, and denoted by “Ava”. To counter potential biases introduced by the RFNet, we also performed a human inspection which involved experienced advertising annotators assessing the available rate, denoted as “Human Ava”. We calculate the total consuming time (hour) using an NVIDIA H800 GPU. For aesthetic assessment, we randomly select 200 available images with identical products, prompts, and generation seed for different approaches. 50 experienced advertising practitioners are employed to rank the images based on their preferences. We count the percentage of different rankings for distinct approaches. 

Results.
The outcomes highlight our approach's superior performance. Several conclusions could be made:


    
  * From the result in Table <ref>, our RFFT could get an extremely high available rate against other approaches. The same trend of “Ava” and “Human Ava” further demonstrates that RFNet reflects human feedback faithfully.
    
  * As shown in <ref>, RG could greatly increase the rate of available images with multiple attempts. Thanks to the inherited strong ability of our model, it needs a shorter production time, shedding light on our RFFL to provide a reliable and efficient solution. 
    
  * The preference assessment in <ref>.a demonstrates that RFFT could achieve a relative aesthetic quality compared to the original model, and gain a lot over ReFL owing to the utilization of CC regularization.
    

These results underscore our approach's enhanced capability to increase the available rate and the producing efficiency while keeping the visual performance stable. Some bad cases solved by our approach are shown in Fig. <ref>.






 §.§ Comparison with KL Regularization

We explore the impact of the hyper-parameter β on both KL regularization L_KL and our proposed CC regularization L_CC. The results, illustrated in Fig. <ref>.b, reveal a notable trend: as β increases, the available rate significantly decreases under L_KL, highlighting its adversarial nature. Conversely, L_CC demonstrates resilience to increased β values, maintaining a higher available rate. This contrast underscores the effectiveness of L_CC in circumventing adversarial effects.



 §.§ Generalization Performance


To assess the flexibility of our RFFT, we examine the generalized capability of the fine-tuned ControlNet when integrated with various LoRAs <cit.> and diffusion model weights. This is critical as retraining the network for each new LoRA or diffusion model weights combination would be impractical. As depicted in Table <ref>, experiments demonstrate that the ControlNet, once refined, significantly enhances the available rate across different LoRAs and diffusion model weights, including Maji_v6[<https://civitai.com/models/43331/majicmix-realistic>] and SD_v1.5[<https://huggingface.co/runwayml/stable-diffusion-v1-5>]. This generalized ability reduces the need for repetitive training, and underscores broader application of RFFT.






 §.§ Integration with Other Feedback


We further incorporate ImageReward <cit.> whose original goal is to improve aesthetics as an additional feedback, F_IR, during the RFFT process. The outcomes, both quantitative and qualitative, are presented in Table <ref> and Fig. <ref>, respectively. The following observations were made:



  * The inclusion of F_IR, and the combination of F_IR and F_AC, significantly enhance aesthetic appeal. However, these often result in excessive detail in the backgrounds, leading to obscured product features, which are undesirable for advertising purposes. Moreover, the combination of F_IR and F_AC results in collapsed textures despite achieving a high available rate.


  * Both F_AC+L_CC and F_IR+F_AC+L_CC configurations achieve high available rates. However, comparing these two, F_IR does not significantly affect background aesthetics. This suggests that while F_IR attempts to modify the background condition, L_CC aims to maintain it. This side-by-side corroborates that our proposed L_CC keeps the background visual quality unchanged, which is particularly important for our RFFL process.








§ CONCLUSION


In conclusion, our study addresses the critical need for reliable advertising image generation in e-commerce, where visual appeal directly influences customer engagement and sales. We introduce Recurrent Generation with the multi-modal RFNet, which faithfully reflects human feedback from our extensive RF1M dataset, establishing a foundation for creating a substantial number of available images for advertising. Our RFFT significantly enhances the generated images available rate, thereby boosting production efficiency. Moreover, our innovative Consistent Condition regularization strikes a balance between a high available rate and aesthetic quality. Ultimately, our work automates creative processes, reduces costs, and improves the customer experience, underscoring the potential of AI-driven tools to revolutionize e-commerce.

splncs04




§ SUPPLEMENTAL MATERIAL


In this supplemental material, we first describe the construction of the RF1M dataset (<ref>). The Recurrent Generation is described in <ref>. We then detail the human-involved experiments (<ref>) and address our responsibilities towards human subjects (<ref>). The detailed configuration for image generation is outlined in <ref>, followed by additional visualization results in <ref>. Finally, we discuss our ethical concerns (<ref>) and future work (<ref>).



§ DATASET CONSTRUCTION




 §.§ Annotation Guidance


During the RF1M annotation process, annotators are provided with the original product images, product captions, and generated images along with the following instructions:

Based on the product image and caption, determine the category of the advertising image:



  * Space Mismatch. Image where the product and background have inappropriate spatial relations contrary to physical laws, such as a part of the product is floating.

  * Size Mismatch. Discrepancies between the product size and its background violate common sense, , a massage chair appears smaller than a cabinet.

  * Indistinctiveness. Image where the product fails to stand out due to background complexity or color similarities, and you cannot clearly figure out which or where the product is.  Additionally, this confusion might lead to misconceptions about the product itself, mistakenly suggesting the inclusion of extra freebies.

  * Shape Hallucination. Background that erroneously extends the product shape, adding elements like pedestals or legs, and giving you incorrect perceptions about the product.

  * Available. Image deemed available for advertising purposes, not falling into any of the above categories.

* If the image belongs to several types simultaneously, , the product has an inappropriate size and extended shape, annotate it with the most significant issue.

And some examples to be annotated are shown in Fig. <ref>





 §.§ More Examples in Dataset

All samples in the RF1M dataset are generated using the approach outlined in paper, created with the original diffusion model before fine-tuning. We showcase additional examples of both available and unavailable generated images in Figs. <ref> and <ref>. For brevity, auxiliary modalities such as product captions, depth images, salience images, and prompts are not shown. The dataset includes a broad variety of products and background scenes with a strong visual appeal.



§ RECURRENT GENERATION


We show our Recurrent Generation strategy in <ref>.





§ HUMAN-INVOLVED EXPERIMENTS


For human availability inspection, the annotators who involved in annotating the RF1M dataset are engaged. They followed the same instructions and data format as during dataset construction (<ref>) to mark whether the generated images were available for advertising use.

For human preference assessment, we generated advertising images using different approaches with the same seed. We combined them with the product image in a random sequence to prevent visual fatigue, as shown in <ref>. Each advertising practitioner ranked them based on personal preference, providing a final ranking like “1423”.



§ RESPONSIBILITY TO HUMAN SUBJECTS


We hired annotators and practitioners to obtain annotations and feedback. We have conducted a thorough review to ensure that the dataset did not include personally identifiable information or offensive content.









§ DETAILED CONFIGURATION


We detail our configuration for image generation in Table <ref>.








§ MORE VISUALIZATION RESULTS




 §.§ Image Collapse


In <ref>, we display collapsed images generated by ReFL. Despite varying product images and prompts, the outputs are uniformly degraded and filled with strange textures, similar to adversarial samples <cit.>. Thus the direct gradient backpropagation without regularization can be interpreted as training the diffusion model to produce images that deceive the RFNet.





 §.§ Generalization Results

Fig. <ref> shows the consistency of background impressions while the available rate improves with different LoRA/diffusion models. 





 §.§ Integration with Other Feedback


We present further examples with different feedback combination strategies in <ref>. Integrating F_IR without L_CC leads to overly detailed backgrounds. Combining F_IR and L_CC preserves background aesthetics, highlighting the role of L_CC to maintain the text prompt conditions in backgrounds.





 §.§ Comparison with Other Refining Approaches


Fig. <ref> and Fig. <ref> showcase advertising images generated by different approaches. Our approach yields images with a high available rate for advertising purposeS while maintaining appealing visuals.








§ ETHICAL CONCERNS


Regarding image manipulation and automated advertising image generation, there's a risk of producing content that could be unethical or illegal, such as violating individual portrait rights or creating discriminatory content. Thus, these technologies require stricter oversight. In our scenario, we carefully select all product images to ensure neutrality and compliance with advertising standards. The prompts for background generation are also meticulously crafted to produce content that is neutral and meets regulatory requirements.

As for our RF1M dataset, all product information is sourced from an e-commerce platform and has undergone pre-review by both the merchants and the platform. The prompts for generating advertising images are created by professional advertising practitioners. During the generation process, we use Stable Diffusion's official safety checker to filter out NSFW content. Finally, professionals review and screen the dataset, ensuring it is free from bias or offensive content and complies with relevant local laws.

In terms of future applications, while these technologies offer significant benefits in efficiency and personalization, they also pose risks related to misinformation. It's vital to maintain high transparency levels to address these concerns. This includes clearly labeling AI-generated images, allowing consumers to differentiate between AI-generated and human-created content. Furthermore, developing and adhering to business standards and ethical guidelines is crucial to ensure AI's use in advertising upholds truthfulness and honesty. The automation of creative tasks could also alter the job market in creative industries. Rather than seeing AI as a replacement for human creativity, we envision a collaborative approach where AI serves as a tool for creative professionals, helping the workforce stay competitive and ensuring human creativity remains vital in advertising.



§ FUTURE WORK


Our research currently emphasizes the availability of generated images. Moving forward, we plan to incorporate additional types of feedback to refine diffusion models. For example, click-through rate (CTR) of different images reveals the customers' preferences. Our future work will explore utilizing the CTR to enhance the appeal of generated advertising images.

Additionally, we have found that conventional metrics for evaluating generated images are infeasible for our specific use case. Given that the central focus of advertising image is the identical product, traditional metrics like FID do not provide an accurate assessment of image quality, , different approaches yield similar FIDs despite varying levels of visual attractiveness. For example, ReFL scores an FID of 18.52 compared to our approach's 16.26 in <ref>. Therefore, we intend to develop a more appropriate evaluation metric for assessing the quality of generated backgrounds.



 Paper Title: Voxel Scene Graph for Intracranial Hemorrhage.






Antoine Sanner
Nils Grauhan
Marc Brockmann
Ahmed Othman
Anirban Mukhopadhyay

A. Sanner et al.
Department of Computer Science, Technical University of Darmstadt, Germany 
 
Department of Neuroradiology, University Medical Center Mainz, Germany 

antoine.sanner@gris.tu-darmstadt.de


Voxel Scene Graph for Intracranial Hemorrhage
    
Antoine P. Sanner1,2 0000-0002-4917-9529 ()  Nils F. Grauhan2 Marc A. Brockmann2 Ahmed E. Othman2 Anirban Mukhopadhyay1

    August 5, 2024
=============================================================================================================================



§ ABSTRACT 

Patients with Intracranial Hemorrhage (ICH) face a potentially life-threatening condition, and patient-centered individualized treatment remains challenging due to possible clinical complications. 
Deep-Learning-based methods can efficiently analyze the routinely acquired head CTs to support the clinical decision-making. The majority of early work focuses on the detection and segmentation of ICH, but do not model the complex relations between ICH and adjacent brain structures. 
In this work, we design a tailored object detection method for ICH, which we unite with segmentation-grounded Scene Graph Generation (SGG) methods to learn a holistic representation of the clinical cerebral scene. 
To the best of our knowledge, this is the first application of SGG for 3D voxel images. 
We evaluate our method on two head-CT datasets and demonstrate that our model can recall up to 74% of clinically relevant relations. This work lays the foundation towards SGG for 3D voxel data.
The generated Scene Graphs can already provide insights for the clinician, but are also valuable for all downstream tasks as a compact and interpretable representation.






§ INTRODUCTION



Intracranial Hemorrhage (ICH) is a potentially life-threatening emergency in which rapid recognition and decision on further treatment is crucial for patient survival <cit.>. The 1-year mortality rate is up to 40% in various studies <cit.> and survivors retain significant persistent functional limitations in two thirds of cases <cit.>. Patient-centered individualized treatment decisions remain clinically challenging, especially considering that patient outcome deteriorates rapidly after ICH onset <cit.>. Given that CT imaging remains the main tool for diagnosis and planning, Deep Learning-based solutions can leverage this data to improve ICH patients' care, e.g. reducing the time to diagnosis.

The majority of early Deep Learning work focuses on the detection and segmentation of ICH <cit.>, but completely ignore the related clinical complications. Indeed, hemorrhages can interact with other brain structures. Through hemorrhage expansion or involvement of the ventricular system, ICH can cause severe complications <cit.>, which are therefore additional predictors for patient outcome <cit.>. Likewise, a bleeding-induced midline shift is a strong predictor of poor patient outcome <cit.>. However, not all bleedings are relevant, such as smaller ones that do not interact with other anatomies. As such, the true potential of automated solutions lies in modeling and understanding complex relations between ICH and adjacent brain structures to build a clinical cerebral scene, that can be then used for downstream tasks such as patient outcome prediction.




Scene Graph Generation (SGG), originally introduced for natural scene-understanding <cit.>, has already been applied in surgical data science <cit.> to model surgical procedures in the operating room. SGG techniques can likewise show their potential, when used with voxel data such as 3D CT scans. Such an application to cerebral scene for ICH is shown in Fig. <ref>. Scene Graphs offer a compact and interpretable representation, that goes beyond pure object detection and can already provide insights for the clinician.
Additionally, an advantage of Scene Graphs is built-in explainability for downstream tasks. 


SGG is a two-step process, which first involves localizing relevant objects in an image. Inspired by Retina U-Net <cit.>, we design a hybrid method that leverages both bounding box annotation and semantic segmentation to both detect objects robustly, and to overcome the challenges of 3D object matching (i.e. orientation-dependent aspect ratios, and significant object overlap). Then, relations between pairs of objects are predicted with SGG methods. Inspired by Neural Motifs <cit.>, and Iterative Message Passing <cit.>, we design two variants for relation prediction for 3D voxel data (V-MOTIF and V-IMP). These use Recurrent Neural Networks to build a global context and from which the relation of each bleeding to adjacent brain structures is predicted.

We show that our model can recall up to 74% of clinically relevant relations. Our contributions include:

    
  * To the best of our knowledge, this is the first time Scene Graphs are generated from 3D voxel data. Scene Graphs can already provide insights for the clinician, independently of their added benefits for downstream tasks.
    
    
  * A tailored object detection method[Code available at <https://github.com/MECLabTUDA/VoxelSceneGraph>] for ICH, which outperforms the state-of-the-art nnDetection <cit.>, and which we unite with our voxel-adapted SGG methods to learn a holistic representation of the clinical cerebral scene.





§ METHODOLOGY


In this section, we give insights into how we structured Scene Graphs for ICH. We then introduce our two-stage method for SGG as visualized in Fig. <ref>.







Scene Graph for ICH: The first and most critical step is to define which structures are relevant, as well as a set of relations that can model possible sources of complications. We select the following interactions for our application based on possible clinical complications: 1) midline shifts, 2) blood flow to the ventricle system, and 3) swelling induced asymmetry of the ventricle system. As such, we have three classes of objects ("Bleeding", "Ventricle System" and "Midline"), and three bleeding-induced classes of relations. Additionally, we expect all patients to have exactly one midline. We model the ventricle system as a single object even when it is only sparsely visible, since the presence of blood can have severe complications independently of the location within the system.

Object Detection: Localizing individual bleeding requires a multiscale approach, as they can have vastly different size, shape, and position. The 3D Retina-UNet coupled with a Feature Pyramid Network has already proved its usefulness for 3D medical imaging <cit.>. In particular, this architecture is very flexible and can fully leverage multiscale features. This is crucial as the volume of the bleeding can range from only 0.1^3 to more than 100^3. 
In opposition, the midline and the ventricle system only appear at a single scale, but pose their own set of challenges. 
Indeed, the 3D aspect ratio of the midline will heavily depend on the head's orientation, especially in the axial plane. This both can cause issues with anchor matching and bounding box regression.
In contrast, depending on the swelling or the presence of blood, the ventricle system may appear as multiple fragmented objects. These would need to be detected individually, as there would be a large overlap between the bleeding with the system and the system itself, rendering an accurate object matching impossible. For these reasons, we decide to leverage Retina-UNet's segmentation capabilities by detecting both anatomies from the predicted semantic segmentation. This simple but elegant solution, can solve the previously mentioned challenged, while also ensuring that only one object is predicted per class per image.

Relation Prediction: Once objects are localized in the image, SGG methods focus on building a global context from individual object features (as defined by <cit.>). Inspired by Neural Motifs <cit.>, and Iterative Message Passing <cit.>, we design two variants for relation prediction for 3D voxel data (V-MOTIF and V-IMP). 

V-MOTIF uses bidirectional Long Short-term Memory Networks and iterates over detected objects. We choose to order these objects from top-to-bottom, as related objects in the cerebral scene are very likely to be at the same depth. Sorting by size, may be another viable solution, but may create a bias towards certain relation classes.
In comparison, V-IMP builds a primal object graph, and its dual edge graph, then combines Gated Recurrent Units (GRU) with message passing iteratively to allow for information flow. Since each GRU receives multiple messages at each iteration, learnable weights are used to pool messages while only keeping the relevant information.
Both models finally predict relations from the enriched features.
Usually, these are also used to refine the classification of each object. 
Given the object matching challenges discussed previously, we decide to only use SGG methods for relation prediction. 

Finally, segmentation-grounding <cit.> involves computing the object features with the object segmentation, rather than with its bounding box, to leverage the finer localization. 
Since only a semantic segmentation in computed in the first step, we use it as a proxy by cropping it to the object's bounds and binarizing it based on the object's predicted class. 



§ EXPERIMENTS



In this section, we introduce the base images used and their annotation process. We then describe our evaluation setup for the different stages of our method.





Source Images: For this study, we source two datasets. The first one is the publicly available INSTANCE2022 challenge dataset <cit.>. 
It contains 130 non-contrast head CTs of patients diagnosed with ICH. 10 images with significant streak artifacts were discarded, as they were deemed out-of-distribution. 
Additionally, we select a private cohort for the purpose of external validation. It is constituted of 18 non-contrast head CTs of patients diagnosed with ICH. We choose more severe cases (caused by trauma) compared to the first dataset.
These cases are clinically more challenging, e.g. some patients were already operated on, and we want to test the robustness of our method against distribution shift (see Fig. <ref>). 




Data Annotation: The patient's head position is first manually harmonized in all images. A senior neuroradiologist then segments a label map for each image using 3D Slicer <cit.>, which takes 5 to 30 min per image. 3D bounding box information is automatically extracted from the label maps by computing the bounds of each segmented object.
Currently, one of the main obstacles of Scene Graph application using voxel data is the lack of tools for relation annotation. In particular, the annotator needs to be able to easily scroll through the 3D volume to select individual objects as being part of a relation. We developed an internal tool specifically to this end. We plan to make it open-source in the near future. On average, annotating relations takes under 2 min per image.


Model Training: For bounding box detection, we split the official training images from the INSTANCE2022 dataset <cit.> for training and validation in an 80/20 fashion. The official validation cases are used for in-distribution testing. The private cohort is fully used for external validation. However, not all images contain relations. As such, we only keep cases with relations for training the relation detector, the splits remaining otherwise unchanged. Configuration files with hyperparameters and  data splits will be made available with the code.

Evaluation Metrics: For bleeding detection, we use the well established metrics for object detection, i.e. Average Recall (AR) and mean Average Precision (mAP). We choose a localization threshold of 30% Intersection Over Union (IoU), as it reflects a balance between the clinical need for coarse localization and producing sensible bounding boxes for larger bleeding <cit.>. 
Additionally, since no relation can get predicted when at least one object has not been detected, we also give an upper bound of how many relations can be recalled given the object detected in the first stage. 
For relation prediction, we compute the metrics based on the top-K predictions per image, with again a 30% IoU threshold for object localization. We use the Recall@K (R@K), mean Recall@K (mR@K), and mean Average Precision@K (mAP@K). Given that an image has on average overall 2 relations (up to 7), we compute these metrics for K=8. Our method is evaluated for both Predicate Classification and Scene Graph Generation tasks <cit.>, i.e. respectively predicting relations from ground truth and predicted object localization. The results are averaged over 5 random seeds.






§ RESULTS



In this section, we evaluate our two-stage pipeline, first for object detection and then for relation prediction (Predicate Classification and Scene Graph Generation). Results for ablations of individual method components can be found in the Supplementary Material.



 §.§ Object Detection



Ventricle System & Midline Detection: We first evaluate the performance for detecting the ventricle system and the midline, as these structures are localized using the predicted semantic segmentation. As shown in Fig. <ref>, our solution can detect them robustly and precisely, with a detection rate of 96.4% for both structures in the INSTANCE2022 dataset. For the private cohort, it still has a detection rate of 100% and 83.3% respectively for the ventricle system and the midline.








Bleeding Detection: We then evaluate the detection performance for bleeding separately from the previously mentioned structures. In Table <ref>, we compare our method to state-of-the-art nnDetection <cit.>. Our method significantly outperforms it on both datasets, showing that careful method parametrization can outperform AutoML for this task. 
The private cohort contains proportionally many smaller bleedings, which are not present in the training data and predominantly have no associated relation. In this regard, our method still robustly detects relevant bleedings in the private cohort.






 §.§ Scene Graph Prediction





Predicate Classification: We evaluate our Voxel Neural Motifs (V-MOTIF) and Voxel Iterative Message Passing (V-IMP) models for relation prediction, with and without segmentation grounding and report the results in Table <ref>. 
Given that datasets for natural scene-understanding <cit.> or even surgical data science  <cit.> comprise thousands of scenes, one could assume that a large data quantity is required for training SGG methods. These results show that under 200 scenes are enough to generalize across medical centers for the ICH cerebral scene.





Scene Graph Generation: This experiment reflects the true performance of the entire prediction pipeline (Table <ref>). Qualitative results are shown in Fig. <ref>. Overall, the models recall relevant relations, with clinically interpretable information for decision support in treatment planning. However, this experiment shows that our complete method already offers a satisfying performance for in-distribution data.  










§ CONCLUSION


The clinical scene for Intracranial Hemorrhage (ICH) goes beyond detecting bleedings, as these can interact with adjacent brain structures, and potentially cause severe complications.
We introduce a method for structured representation learning and tailored for ICH. To the best of our knowledge, this is the first application of Scene Graph Generation (SGG) for 3D voxel data. 
We evaluate our method on two datasets and we demonstrate that our model can recall up to 74% of clinically relevant relations.
This work paves a path towards SGG for 3D voxel images and modeling holistic anatomical scenes.
The generated Scene Graphs can be used as a compact and interpretable representation for all downstream task (e.g. patient outcome prediction) and offer a strong decision support for patient-centered treatment.


§ COMPLIANCE WITH ETHICAL STANDARDS


This study was performed in line with the principles of the Declaration of Helsinki. The retrospective evaluation of imaging data from the University Medical Center Mainz was approved by the local ethics boards (Project 2021-15948-retrospektiv). Ethical approval was not required, as confirmed by the license attached with the open access data.






  §.§.§ 
 Antoine P. Sanner and Nils F. Grauhan equally contribute to this paper.



  §.§.§ 

The authors have no competing interests to declare that are relevant to the content of this article.












splncs04




 Paper Title: Spatial Transformer Network YOLO Model for Agricultural Object Detection.





Spatial Transformer Network YOLO Model for Agricultural Object Detection


 



    Yash Vivek Zambre^1, Ekdev Rajkitkul^2, Akshatha Mohan^1, Joshua Peeples^1
 ^1Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX, USA

	^2Department of Computer Science and Engineering, Texas A&M University, College Station, TX, USA

		yashzambre@tamu.edu, ekdev@tamu.edu, akshatha.mohan@tamu.edu, jpeeples@tamu.edu
    
=========================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Object detection plays a crucial role in the field of computer vision by autonomously identifying and locating objects of interest. The You Only Look Once (YOLO) model is an effective single-shot detector. However, YOLO faces challenges in cluttered or partially occluded scenes and can struggle with small, low-contrast objects. We propose a new method that integrates spatial transformer networks (STNs) into YOLO to improve performance. The proposed STN-YOLO aims to enhance the model's effectiveness by focusing on important areas of the image and improving the spatial invariance of the model before the detection process. Our proposed method improved object detection performance both qualitatively and quantitatively. We explore the impact of different localization networks within the STN module as well as the robustness of the model across different spatial transformations. We apply the STN-YOLO on benchmark datasets for Agricultural object detection as well as a new dataset from a state-of-the-art plant phenotyping greenhouse facility. Our code and dataset are publicly available [<https://github.com/Advanced-Vision-and-Learning-Lab/STN-YOLO>].




Spatial transformer network, Object detection, YOLO, plant phenotyping






§ INTRODUCTION

Plant phenotyping is critical for crop improvement <cit.>, yield optimization <cit.>, and sustainable practices <cit.>. Artificial intelligence (AI), particularly object detection algorithms, has transformed plant phenotyping, enhancing efficiency and performance <cit.>. The You Only Look Once (YOLO) has been used effectively in various Agricultural applications such as pest detection <cit.>, crop disease detection <cit.>, and crop harvesting <cit.>. Despite the vast number of use cases, YOLO has some limitations due to various spatial transformations <cit.>. Spatial transformer networks <cit.> are an approach to improve an artificial neural network's robustness to spatial transformations. We propose to integrate STNs with the YOLO model to incorporate spatial invariance. The STN applies a learnable affine transformations to the images that will help with object detection. The STN-YOLO model demonstrates spatial invariance and outperforms the baseline YOLO model on several Agricultural benchmark datasets.



Additionally, we present a new, high quality, annotated dataset of various plants to advance the study of agriculture plant detection and phenotyping. The new Plant Growth and Phenotyping (PGP) dataset differs from other available image datasets in the following aspects: (1) multi-spectral images captured across varying heights; (2) challenging images of multiple crops across varying illumination conditions; (3) precise annotations assisted with the use of Segment Anything model <cit.>, (4) large size and shape variations of plants. The features of the PGP dataset present new challenges for agricultural object detection. We perform extensive experiments on the proposed database and new model (STN-YOLO) for object detection. The key contributions of our work are the following:

    
  * Integration of STN within YOLO for improved object detection performance
    
  * New benchmark dataset for plant object detection.



§ RELATED WORK



 §.§ Object Detection Models

Object detection is a prominent challenge in computer vision, where researchers use deep learning to boost performance <cit.>. Two-stage detectors, such as Faster R-CNN <cit.>, involve distinct stages with a region proposal network (RPN) and region of interest (ROI) pooling for candidate bounding boxes <cit.>. In contrast, single-stage detectors such as single-shot detectors (SSD) predict bounding boxes directly by utilizing grid boxes and anchors, as described in <cit.>. Notably, YOLO has emerged as a prominent example of a single-stage detector. The advancements of CNNs architectures introduces innovations such as anchor boxes to enhance object detection <cit.>. Key advancements include R-CNN, integrating region proposals with CNN using support vector machine (SVM) classification and bounding-box regression <cit.>. However, R-CNN incurs substantial computational costs and information loss. Fast R-CNN extends this with ROI pooling and proposal refinement, improving test speed and algorithm precision <cit.>. Faster R-CNN further reduces computational requirements by introducing the RPN for regional proposals, selecting anchors based on specific criteria for outstanding recognition precision <cit.>. Ongoing advancements focus on speed optimization, including anchor-free detection <cit.>.



 §.§ Spatial Transformer Network (STN)


STN is a module with differentiable properties that performs learnable spatial transformation on input feature maps <cit.>. The transformation is dependent on the specific input, generating a singular output feature map. In the case of multi-channel inputs, an identical transformation is applied on each channel. The STN module is comprised of three components, as illustrated in the Figure <ref>. The first component is a localization network, which passes input feature maps through several hidden layers to generate the parameters for a learnable affine transformation that is applied to each feature map. Subsequently, the predicted transformation parameters θ are used to construct a sampling grid—a set of points indicating where the input feature maps should be sampled to generate the transformed output. Finally, the feature maps and the sampling grid serve as inputs to the sampler that generates the output map sampled from the input at the specified grid points.

Localization Networks The localization network (f_loc) takes the input feature maps 𝐗∈ℝ^H × W × C with width W, height H, and C channels. The features extracted by the localization network are then passed into a fully connected layer that outputs θ, the parameters of the transformation T_θ to be applied to each feature map: θ = f_loc(X). The size of θ can vary depending on the transformation type that is parameterized <cit.>. The localization network function f_loc() can take any form, such as a fully-connected network or a CNN such as ResNet18 <cit.>, but should include a final regression layer (i.e., fully connected layer) to produce the transformation parameters θ (e.g., affine transformation have six parameters).

Parameterized Sampling Grid
To perform a transformation on the input feature map, each resulting “pixel" is determined by applying a sampling kernel centered at a specific location in the input feature map. The term “pixel" in this context refers to an element of a general feature map, not necessarily an image. In a general context, the resulting “pixels" are positioned on a regular grid G = {G_i} of pixels G_i = (x^t_i, y^t_i), creating an output feature map V ∈ℝ^H' × W' × C. Here, H' and W' represent the height and width of the grid, respectively, and C is the number of channels, which is same in the input and output feature maps.

For clarity of exposition, assume that T_θ is a 2D affine transformation A_θ. In the affine case, the point-wise transformation is shown in Equation <ref>:

    [ x_i^s; y_i^s ]
    = T_θ(G_i) = A_θ[ x_i^t; y_i^t;     1 ]
    = 
    [ θ_11 θ_12 θ_13; θ_21 θ_22 θ_23 ][ x_i^t; y_i^t;     1 ]



In this scenario, (x_i^t, y_i^t) denotes the target coordinates of the regular grid in the output feature map, and (x_i^s, y_i^s) represents the source coordinates in the input feature map that define the sample points. The relationship between these coordinates is determined by the affine transformation matrix A_θ <cit.>. The STN is self-contained module that is versatile and can be seamlessly integrated at any point in an artificial neural network architecture. The inclusion of an STN at the beginning of a CNN enables the network to learn how to dynamically transform input images to account for spatial variations.



§ METHOD




 §.§ Plant Growth and Phenotyping Dataset



The Plant Growth and Phenotyping (PGP) dataset is a publicly available, high-resolution (512 × 512 pixels), multi-spectral collection of plant images captured from a state-of-the-art greenhouse. PGP consists of 1137 images from different crops: 442 corn, 246 cotton, and 449 rice plants. The dataset includes images taken from various heights of the same plants. 



The plant phenotyping facility consisted of a camera mounted on a robotic arm, capturing side and top views of the plants (only side view images are used in this work). The Multispectral Imaging System (MSIS-AGRI-1-A), integrating an MSC2-AGRI-1-A snapshot multispectral camera and a 4-channel LED illuminator, was used to collect images across four spectral bands: 580 nm (Green), 660 nm (Red), 730 nm (Red Edge), and 820 nm (Near Infrared). 

For this work, the Red, Red Edge, and Green bands were combined to create pseudo “RGB" images (as shown in Figure <ref>) compatible with pre-trained machine learning models. The individual bands were superimposed, and min-max normalization was applied to the final image. These plant images were captured between 2022 and 2024 in controlled greenhouse environments. Data augmentation techniques (e.g., rotation, cropping) can be applied using frameworks like Roboflow <cit.> to expand the dataset. The same framework is used for labeling, a two-step process involving auto-generation of initial labels using the Segment Anything Model (SAM) <cit.>, followed by manual refining of the auto-generated labels.




 §.§ STN-YOLO MODEL
    For the proposed STN-YOLO model, the STN module is first integrated in the beginning of the YOLO model to account for spatial transformations in the input images. After the images passes through STN module, the YOLO model is then used to perform the task of object detection. As stated in Section <ref>, the STN has can use any backbone for the localization networks. The localization network within the STN module comprises various convolutional layers, linear layers, and activation functions. 


Localization in STN model 
The STN models have the localization network which takes the input feature maps with width W and height H and C channels and outputs θ which are the transformation parameters. The localization network is selected by the user and can vary from a shallow or deeper network. The performances of different localization network are compared to evaluate if a shallow or deep, pre-trained model will lead to better object detection performance quantitatively (e.g., accuracy, precision, and recall) and computationally. The shallow network is comprised of one convolution layer with max pooling and ReLU activation function was used in contrast to the deep pretrained models. Following the convolution layer in the shallow network, an adaptive average pooling layer was used to aggregate the spatial resolution of the feature maps to a desired size (e.g., 28 × 28).

YOLO Backbone The output of the STN module is then given to the YOLO model where, P1, P2, P3, P4, and P5 typically refer to different levels or stages in the feature pyramid and these are the different convolutional layers as shown in the Figure <ref>. Output features are obtained by extracting features at different scales from the input image  to detect objects of various sizes and capture both fine-grained details and global context. P1 corresponds to the finest level with the highest resolution, while P5 is the coarsest level with lower resolution but capturing more context. The head in the YOLO model usually refers to the part of the network responsible for making predictions. The head takes features from multiple pyramid levels and produces predictions for object classes, bounding box coordinates, and other relevant information. 

Loss in YOLOv8
A two term loss function is used during training. The objective function consists of two components: classification loss (CLS), which penalizes errors in predicting object classes, and bounding box (BBOX) loss, which penalizes errors in predicting the coordinates of the bounding boxes around objects. Combining these losses helps train the model to simultaneously improve both classification and localization aspects of object detection. Class imbalance in single-class detection refers to the discrepancy between detection target objects and the background, typically considered the negative class. Addressing this imbalance is crucial for improving detection performance. Bounding box loss in YOLOv8 uses the complete intersection over union (CIOU) loss <cit.>, which considers box overlap and aspect ratio differences, enhancing regression performance. Distributed focal loss (DFL) <cit.> effectively tackles class discrepancy by dynamically adjusting the loss for each class during training to counteract biases caused by class imbalances.







§ EXPERIMENTAL RESULTS AND DISCUSSION




 §.§ Experimental Setup

 Three experimental runs of random initialization were used for each model and the YOLO backbones were pretrained on the COCO dataset. We used the AdamW optimizer with an initial learning rate of 0.002. The optimizer was configured with distinct parameter groups, each with specific weight decay values assigned to all parameters. The batch size for all experiments was set to 16. We used 100 epochs with early stopping (patience of 50) for each model. All the experiments were performed using the Ultralytics <cit.> framework and performed on an NVIDIA A100 GPU.

In our study, we used various benchmark datasets to assess the effectiveness of our proposed plant detection models.
The GlobalWheat2020 dataset contains 4,000 images, with 3,000 for training and 1,000 for testing, focusing on wheat detection.
The PlantDoc dataset comprises 2,569 images, with 2,330 for training and 239 for testing, addressing plant identification across varying environments.
The MelonFlower dataset includes 288 images, with 193 for training and 95 for testing, specifically targeting melon flower identification. These datasets collectively provide a robust evaluation framework for assessing model generalization across diverse plant species and environments. 





 §.§ Ablation Studies using PGP Dataset


Impact of Localization Networks
In Table <ref>, we outline the performance metrics of various localization networks, emphasizing the comparable quantitative results achieved by both the shallow localization network and deep pre-trained models.  Both the shallow and deep networks used a global average pooling layer after the convolutional layers. Based on these findings, we opted for the shallow network as the localization network in the STN-YOLO model. This decision stemmed from the observed effectiveness of the shallow network in extracting features suitable for learning affine transformations comparable to the capabilities of the pre-trained models (based on similar quantitative measures). The shallow network and the pretrained networks were finetuned  with the baseline YOLO model. The number of learnable parameters for the deeper networks, VGG16 and ResNet18 (approximately 82 million and 49 million respectively), were greater than the shallow network (approximately 5 million) for marginal improvements in performance hence the shallow network was chosen for the remaining experiments.




Impact of Spatial Information
To assess the impact of retaining more spatial information in the STN-YOLO model, modifications were applied to the last layer of the shallow localization network. Specifically, the adaptive average pooling layer, was evaluated for various spatial sizes of 1 × 1, 7 × 7, and 28 × 28 as shown in Table <ref>. This investigation aims to evaluate the model's sensitivity to variations in spatial information preservation and identify the optimal configuration for improved performance in object detection tasks. 
We wanted to look at no spatial information (1 × 1) and retaining some spatial information (7 × 7, 28 × 28). The results demonstrated that retaining the most amount of spatial information (28 × 28) resulted in the best object detection performance. If the feature maps have more spatial information, then there is more context that the model will be able to use for maximizing performance. However, as the spatial resolution is increased, so does the computational cost as the number of parameters will increase. 



Model Analysis using Explainable AI The proposed model is better on average at reducing the detection of false positives, as the precision of the STN-YOLO model is better than the baseline YOLO model as shown in Table <ref>. The STN-YOLO model also exhibits improved recall and mean average precision (mAP), which demonstrates a better object detection model as illustrated in Figure (<ref>-<ref>), where the baseline YOLO model fails to eliminate the bottom right vase area in comparison to the proposed STN-YOLO model. In addition to these metrics from Table <ref>, we also provide insights from Eigen class activation maps (EignenCAMs) <cit.> as seen in Figure (<ref>-<ref>). These EigenCAMs offer valuable visualization for model explainability, as they highlight significant aspects of the images that contribute to the model's predictions. The results support the hypothesis that incorporating the spatial invariance in the input image can result in better object detection both qualitatively and quantitatively as the STN-YOLO emphasizes more of the plant area.





Augmentations Testing One of the most important set of experiments performed was the testing  of models on the unseen augmented data as shown in Table <ref>. To do this, no data augmentation was added to the training dataset, but data augmentations were performed on the test dataset to evaluate the robustness of the STN-YOLO model across various transformations. The augmentations used on the test dataset were random cropping (15% zoom), shear (± 10^∘ horizontal and ± 10^∘ vertical), and rotation (between ± 10^∘) to simulate image conditions that may happen in the facility. We observed that the proposed STN-YOLO model gave better quantitative results in case of rotation and shear for PGP dataset compared to the baseline YOLO model. This result is intuitive as rotation and shear are affine transformations that can be learned by the STN-YOLO model. However, in the cases where cropping is used, the performance is comparable to the baseline YOLO method as cropping is not an affine transformation. Furthermore, for the combinations of the different augmentations (e.g., rotation with shear, rotation with crop), we observed that for most of the data augmentation cases (six out of eight), the STN-YOLO model performs better on average than the YOLO baseline for the precision values indicating that the STN-YOLO was effective at reducing the number of false positives by incorporating the added spatial invariance properties of the STNs.







 §.§ Benchmark Datasets Results


GlobalWheat2020 Dataset The GlobalWheat2020 dataset <cit.> is a comprehensive Agricultural dataset containing images of wheat heads sourced from various countries. From the results in Table <ref>, the  STN-YOLO model has a higher average value for most metrics (except accuracy), but the baseline YOLO model is comparable to the average value when you take the standard deviation in comparison. We can also qualitatively look at this observation from Figure <ref> where the number of detections by YOLO in the GlobalWheat2020 dataset are less as compared than the detections of STN-YOLO.
For the GlobalWheat2020 dataset, the images consisted of “zoomed"-in views of the wheatheads. In this case, the spatial invariance would not impact the object detection results and this is indicated by the slight improvements in the average object detection metrics (except for accuracy) shown in Table <ref>.

PlantDoc Dataset The PlantDoc dataset <cit.> is derived from the Plant Village dataset (PVD) <cit.>. The dataset was created using an automated system leveraging GoogleNet <cit.> and AlexNet <cit.>. Interesting observations can also be seen from the Table <ref> such as unlike the GlobalWheat2020 dataset, there is an increase in all the object detection metrics in the STN-YOLO model as compared to the baseline model. Particularly, the precision for STN-YOLO model is higher than the baseline model indicating the model reduced the number of false positives detected as noted for the PGP dataset. Qualitatively, we can see that in Figure <ref> where the YOLO detects the leaf correctly but wrongly classifies the leaf as tomato early blight leaf. However, the STN-YOLO detects and correctly classifies the leaf as Tomato Septoria. In the PlantDoc dataset, the emphasis remains on various plants and zoomed-in images. However, spatial invariance remains insignificant due to the absence of pre-translated or rotated input images. Despite this, spatial invariance could potentially offer some benefits with this dataset, particularly as certain images provide a more “zoomed-out" perspective of the plants compared to the GlobalWheat2020 dataset.

MelonFlower Dataset The MelonFlower dataset <cit.> is dataset available on Roboflow and the images were captured in a greenhouse (similar to the PGP dataset). This dataset was used to demonstrate the model's capability in detecting small objects (e.g., flowers) and different viewing geometries of the plants. As seen from Table <ref>, there is a significant difference in the precision values with the STN-YOLO model as compared to the baseline model. The precision metrics reveal a substantial improvement, showcasing the model's enhanced ability to accurately detect objects.  Visually, we can see this difference in the Figure <ref> where in the YOLO detections of the Melonflower dataset, the model detects the melon flower with an IOU of 0.50 and the STN-YOLO detections of the Melonflower dataset has a better improved detection with an IOU value of 0.60. This difference not only emphasizes the effectiveness of the STN-YOLO model but also underscores its potential for applications where precision is a critical performance indicator. Since the MelonFlower dataset had the most spatial variance among all datasets evaluated in this work, the STN-YOLO model exhibited notably improved performance, as evidenced by the precision values and other object detection metrics presented in Table <ref>.



§ CONCLUSION

This work focused on integrating STN with YOLO, creating the STN-YOLO model to address spatial invariance challenges. The model aimed to enhance plant image detection quality and can be used for downstream applications such as phenotypic feature extraction. Results demonstrated that STN improved model robustness and reduced number of false positives in the datasets as indicated by the higher precision scores. The STN-YOLO model improved performance on benchmark datasets, showcasing potential in handling real-world spatial transformations. Future work includes integrating STNs into other object detection models (e.g., future versions of YOLO <cit.>), developing new objective functions to improve the learning of the model with STNs, expanding our PGP dataset to include more images across multiple crops with different image conditions (e.g., illumination), and incorporating the near infrared channel. 



§ ACKNOWLEDGEMENT


This material is based upon work supported by Texas A&M AgriLife. Portions of this research were conducted with the advanced
computing resources provided by Texas A&M High Performance Research Computing.

IEEEtran




 Paper Title: Leveraging Adaptive Implicit Representation Mapping for Ultra High-Resolution Image Segmentation.





 










Ziyu Zhao, Xiaoguang Li, Pingping Cai, Canyu Zhang, Song Wang




University of South Carolina, USA 

{ziyuz,xl22,pcai,canyu}@email.sc.edu, songwang@cec.sc.edu

Leveraging Adaptive Implicit Representation Mapping for Ultra High-Resolution Image Segmentation
    
Ziyu Zhao
Xiaoguang Li
Pingping Cai
Canyu Zhang
Song Wang
    August 5, 2024
================================================================================================




§ ABSTRACT 
  Implicit representation mapping (IRM) can translate image features to any continuous resolution, showcasing its potent capability for ultra-high-resolution image segmentation refinement. Current IRM-based methods for refining ultra-high-resolution image segmentation often rely on CNN-based encoders to extract image features and apply a Shared Implicit Representation Mapping Function (SIRMF) to convert pixel-wise features into segmented results. Hence, these methods exhibit two crucial limitations. Firstly, the CNN-based encoder may not effectively capture long-distance information, resulting in a lack of global semantic information in the pixel-wise features. Secondly, SIRMF is shared across all samples, which limits its ability to generalize and handle diverse inputs.
  
  

  To address these limitations, we propose a novel approach

  that leverages the newly proposed Adaptive Implicit Representation Mapping (AIRM) for ultra-high-resolution Image Segmentation. 

  Specifically, the proposed method comprises two components: (1) the Affinity Empowered Encoder (AEE), a robust feature extractor that leverages the benefits of the transformer architecture and semantic affinity to model long-distance features effectively, and (2) the Adaptive Implicit Representation Mapping Function (AIRMF), which adaptively translates pixel-wise features without neglecting the global semantic information, allowing for flexible and precise feature translation.

  We evaluated our method on the commonly used ultra-high-resolution segmentation refinement datasets, , BIG and PASCAL VOC 2012. The extensive experiments demonstrate that our method outperforms competitors by a large margin. The code is provided in supplementary material.
  







§ INTRODUCTION



Image segmentation, the process of partitioning pixels of an input image into distinct regions, has shown notable progress. While prior methods excel at the low-resolution segmentation task <cit.>, addressing the task of ultra-high-resolution segmentation still poses a persistent challenge due to substantial computational demands.



To address the ultra-high-resolution image segmentation problem, a commonly employed strategy involves refining the segmentation outcomes produced by low-resolution models. Following this strategy, <cit.> proposes a refinement network with a cascade scheme, where the intermediate refinement outcomes are iteratively upsampled to higher resolutions.


Despite achieving surprising results, this progressive process is quite time-consuming.

To improve the computational efficiency and reduce the inference time, CRM <cit.> draws inspiration from LIIF <cit.> and uses Implicit Representation Mapping (IRM) to achieve the refinement of ultra-high-resolution image segmentation.

While exhibiting promising performance, existing IRM-based methods, , CRM and LIIF, typically rely on a CNN-based encoder to extract image features and a Shared Implicit Representation Mapping Function (SIRMF) to translate the pixel-wise features, 

overlooking two critical limitations: (1) the CNN-based encoder may struggle to effectively capture long-distance information, leading to a lack of global semantic information in pixel-wise features. (2) SIRMF is shared across all samples, limiting its generalization capability to handle diverse inputs.

As a result, these two limitations significantly impact the performance of current IRM-based methods (see the yellow circle in fig:fig1).


To address these limitations, our work begins by conducting comprehensive experiments to examine the impact of receptive fields on IRM. We observe that larger receptive fields significantly enhance the performance of IRM. Additionally, we analyze the weaknesses of the existing SIRMF and find that it lacks the generalization capability to handle diverse inputs. Based on these observations, we introduce a novel refinement approach that Leveraging Adaptive Implicit Representation Mapping (AIRM) for Ultra High-Resolution Image Segmentation.

The proposed network comprises two components: the Affinity Empowered Encoder (AEE) and the Adaptive Implicit Representation Mapping Function (AIRMF). The AEE, responsible for feature extraction, leverages the advantages of transformer architecture and semantic affinity to model long-distance features effectively. Unlike using shared mapping parameters, AIRMF's parameters are predicted by a separate network based on the entire feature extracted by AEE. This enables AIRMF to adaptively translate pixel-wise features without neglecting global semantic information, allowing for flexible and precise feature translation.

As a result, the proposed method can extract pixel-wise features with a significantly larger receptive field and translate them into the segmentation result in an adaptive manner, collaborating with global semantic information.

Our main contribution is the following.


    
  * We conduct comprehensive experiments to explore the impact of receptive fields on IRM and analyze the limitations of the existing SIRMF. The experimental results demonstrate two key findings: (1) Larger receptive fields significantly enhance the performance of IRM, and (2) the existing SIRMF lacks the generalization capability to handle diverse inputs and tends to overlook the global semantic information.

    
  * We introduce a novel approach named Leveraging Adaptive Implicit Representation Mapping for Ultra-High-Resolution Image Segmentation, which comprises two components: the Affinity Empowered Encoder (AEE) and the Adaptive Implicit Representation Mapping Function (AIRMF). The proposed method can extract pixel-wise features with a significantly larger receptive field and translate them into the segmentation result in an adaptive manner, collaborating with global semantic information.

    
  * The extensive experiment results on multiple ultra-high-resolution segmentation refinement datasets demonstrate that our method outperforms competitors by a large margin.
    







§ RELATED WORK










 §.§ Refining Semantic Segmentation


The progress in semantic segmentation, which aims to assign distinct categories or classes to individual pixels within an image, has been significantly propelled by the achievements of deep neural networks <cit.>. 

However, these methods are primarily focused on tackling the issue of low-resolution image segmentation but fall short of producing high-quality segmentation results. 
Thus,

the task of refining segmentation is introduced to improve the quality of segmentation outcomes for high-resolution (1K∼2K) (HR) or even ultra high-resolution  (4K∼6K) (UHR) images. Unlike low-resolution images, HR and UHR require a deeper understanding of semantic details. 
Conventional techniques rely on graphical models like CRF <cit.> or region growing <cit.> techniques to achieve HR segmentation. But they mainly focus on low-level color boundaries <cit.>, limiting their segmentation performance. 
In contrast, Multi-scale Cascade Networks <cit.> employ a recursive approach to progressively integrate multi-scale information. Nevertheless, such a recursive approach inadvertently introduces more computational burdens. Therefore, a Continuous Refinement Network <cit.> is proposed to improve the efficiency by continually aligning the feature map with the refinement target. However, it tends to overlook the significance of the limited receptive field of the Convolution Operation in the CNN backbone and also lacks the generalization capability to effectively handle specific categories. To address these concerns, we employ a Transformer-based adaptive implicit representation function to mitigate these limitations and improve segmentation accuracy.




 §.§ Transformer in Segmentation




Vision Transformers (ViT) <cit.>, as demonstrated by their remarkable success, is adapted for various computer vision tasks such as semantic segmentation <cit.> and object segmentation <cit.>. 

However, applying plain Vision Transformer (ViT) models directly to segmentation tasks is challenging due to their absence of segmentation-specific heads. To solve this problem, SETR <cit.> reframes semantic segmentation as a sequence-to-sequence prediction task. <cit.> proposes a versatile hierarchical Vision Transformer backbone, known as Swin Transformer (Hierarchical Vision Transformer using Shifted Windows), using shifted windows to efficiently capture long-range dependencies while maintaining scalability. Segmenter <cit.> employs pre-trained image classification models fine-tuned on moderate-sized segmentation datasets for semantic segmentation tasks. 

Instead of ViT, <cit.> utilizes Transformers to generate more integral initial pseudo labels for the end-to-end weakly-supervised semantic segmentation task and proposes an Affinity from Attention (AFA) module to learn semantic affinity from the multi-head self-attention (MHSA) in Transformers.

Similar to <cit.>, we propose to integrate the Transformer architecture into our UHR image refinement segmentation task and leverage the learned semantic affinity from multi-head self-attention to enhance the precision of coarse masks. 






 §.§ Hypernetworks


Hypernetworks or meta-models, as referenced in <cit.>, involve designing models specifically tailored for generating parameters for other models. This parameterization approach not only enhances the model's expressiveness, as highlighted in <cit.>, but it also supports data compression through weight sharing across a meta-model <cit.>. 
Unlike conventional deep neural networks, where the weights remain fixed during inference, hypernetworks possess the ability to generate adaptive weights or parameters for target networks <cit.>. We leverage this technique and propose an Adaptive Implicit Representation Mapping Function. Essentially, an additional Convolution Network is responsible for learning the parameters of the implicit function in the segmentation network,

enhancing its adaptability to a variety of high-resolution image segmentation tasks. 












§ DISCUSSION AND MOTIVATION





In this section, we conduct several comprehensive experiments to explore the impact of receptive fields on IRM 
(see receptive_fields) and analyze the limitation of the existing SIRMF (see limitation_ip). 
We observe that (1) larger receptive fields significantly enhance the performance of IRM, and (2) the existing SIRMF lacks the generalization capability to handle diverse inputs and tends to overlook global semantic information. Building on these observations, we introduce a novel approach named Leveraging Adaptive Implicit Representation Mapping for Ultra-High-Resolution Image Segmentation. This approach utilizes transformer architecture and semantic affinity to effectively model long-distance features, enabling adaptive translation of pixel-wise features while preserving global semantic information.




 §.§ The Impact of Receptive Fields on IRM



Image encoder. To investigate the impact of receptive fields on IRM, we construct a straightforward image encoder, ϕ, comprising only several convolutional layers, followed by ReLU and batch normalization operations. We then utilize the encoder ϕ to extract features 𝐅∈ℝ^h × w × D from the input image 𝐈 and coarse segmentation mask 𝐌,



    𝐅 = ϕ(𝐈, 𝐌).




SIRMF. Following <cit.>, the SIRMF is implemented by an MLP network, denotes as f_θ, and the refined segmentation result 𝐌_q at location q, can be represented as



    𝐌_q = ∑_p ∈𝒩_qS_p/∑S_p·f_θ(z_p, p - q),



where f is the mapping function, θ is the trainable parameter of f,  𝒩_q are neighbor locations around q, and z_p is the pixel-wise feature vector related to location p. The aggregation weights, area value of S_p, are calculated based on the relative coordinate offset between q and p.



We train the image encoder ϕ with various kernel sizes and the SIRMF f_θ on the high-resolution image segmentation dataset. Subsequently, we evaluate the performance based on the Intersection over Union (IoU) and illustrate the results in fig:dis (a) (see the blue line). We can observe a substantial performance improvement in the segmentation results when enlarging the kernel size. This highlights that larger receptive fields significantly enhance the performance of IRM in the context of image segmentation tasks. Therefore, in this study, we propose using a transformer-based architecture enhanced with semantic affinity as the image encoder, moving away from the CNN-based encoders commonly used in previous research, , CRM <cit.> and LIIF <cit.>.


 







 §.§ Limitation of The existing SIRMF



To further analyze the limitations of the current SIRMF, we randomly select three categories of images from the segmentation dataset. Subsequently, we conduct the following experiments: 

Exp.1 We utilize the pre-trained image encoder ϕ from receptive_fields to extract image features. We train a SIRMF on the entire selected image set and then evaluate the performance of the SIRMF on each category separately (see fig:dis (b) blue bar). 

Exp.2 We use the same encoder as in Exp.1 to extract image features. But we train three SIRMF, each dedicated to one of the selected categories, and then evaluate the performance of each SIRMF on its corresponding category (see fig:dis (b) yellow bar). 

From fig:dis (b) we can observe that training three SIRMF separately for each category yields better performance compared to training one SIRMF across all three categories. This observation indicates that the existing SIRMF lacks the generalization capability to effectively handle specific categories and tends to overlook global semantic information.





§ METHODOLOGY



Building upon the observations in dis_moti, in this section, we introduce a novel approach that leverages adaptive implicit representation mapping for ultra-high-resolution image segmentation to effectively address the previous limitations, as shown in pipeline. Specifically, the proposed method AIRM comprises two components: the Affinity Empowered Encoder (AEE) (see encoder) and the Adaptive Implicit Representation Mapping Function (AIRMF) (see mapping). The AEE is responsible for effectively modeling long-distance features by taking advantage of the transformer architecture and semantic affinity. The AIRMF is responsible for adaptively translating pixel-wise features without neglecting the global semantic information, allowing for flexible and precise feature translation.

 




 §.§ Affinity Empowered Encoder























As discussed in dis_moti, larger receptive fields significantly enhance the performance of IRM. Therefore, instead of employing a CNN-based image encoder like CRM <cit.> and LIIF <cit.>, we propose an affinity-empowered image encoder that takes advantage of the transformer architecture and semantic affinity <cit.>.

Specifically, given the RGB image 𝐈∈ℝ^H × W × 3 and the corresponding coarse segmentation mask 𝐌∈ℝ^H × W × 1, we employ the transformer-based encoder ω to extract features 𝐅, which can be represented as follows:


    𝐅, 𝐀𝐓𝐓 = ω(𝐈, 𝐌),


where 𝐀𝐓𝐓∈ℝ^hw × hw × c denotes the attention map produced by ω and hw represents the flattened spatial dimensions and c is the number of attention heads. Note that the coarse segmentation mask is obtained by applying an existing segmentation method such as <cit.>.




To further empower the capability of the transformer-based encoder, we follow <cit.> to learn the affinity from the attention map 𝐀𝐓𝐓 to intentionally impose constraints on the encoder ω during training. 

The self-attention mechanism functions as a directed graphical model, where the attention 𝐀𝐓𝐓 is typically coarse (see pipeline). The affinity matrix is expected to be symmetric since nodes with identical semantics should be equal.

Therefore, following <cit.>, we obtain the predicted affinity 𝐀𝐟𝐟_pre∈ℝ^hw × hw (see pipeline) by feeding the sum of 𝐀𝐓𝐓 and its transpose into an MLP network, , φ, which can be represented as: 

    𝐀𝐟𝐟_pre = φ(𝐀𝐓𝐓 + 𝐀𝐓𝐓^⊤).

We expect 𝐀𝐟𝐟_pre to be close to the ground truth affinity 𝐀𝐟𝐟_gt∈ℝ^hw × hw obtained from the ground truth segmentation mask 𝐌̂. Specifically, we resize 𝐌̂ to hw × hw, and the affinity between two coordinates (x_i, y_i) and (x_j, y_j) is assigned as 1 if their classes are identical, and 0 otherwise.













 













 §.§ Adaptive Implicit Representation Mapping Function (AIRMF)

















In limitation_ip, we identified the limitation in the existing SIRMF (<ref>) — it lacks the capacity to effectively generalize across diverse inputs (see fig:dis (b)). To tackle this issue, we introduce SIRMF which can be represented as



    𝐌_q = ∑_p ∈𝒩_qS_p/∑S_p·f_ψ(𝐅)(z_p, p - q),



where ψ(·) denotes a hypernetwork that is used to predict the adaptive mapping parameters.

Instead of directly optimizing the shared parameter θ in (<ref>), we employ ψ to predict it adaptively by inputting the feature 𝐅. Intuitively, the predicted mapping parameter should capture global semantic information from the input feature 𝐅. 

Consequently, AIRMF can leverage both local information from pixel-wise features and global information from the entire image feature. This design allows AIRMF to yield precise segmentation results (see the red points in fig:dis (a) and the red bar in fig:dis (b)).


Assume we aim to generate the weight W^l∈ℝ^n_ in× n_ out and bias b^l∈ℝ^n_ out of the l-th linear layer of f(·). To optimize computational efficiency, we do not directly predict the parameter. Instead, we adopt a strategy similar to <cit.>, predicting two intermediary rectangular  matrices A^l∈ℝ^n_ out^l × r  and B^l∈ℝ^r × n_ in^l  through ψ(·), where r = 20 for all the layers of f(·). The linear modulating projection weight W^l is then modulated by multiplying these matrices as follow, 

    W^l = A^l×B^l.

Bias parameter b^l is directly produced due to its small volume. We streamline the AIRMF in pipeline. 












 §.§ Implementation Details


Network architectures. We employ the Mix Transformer (MiT) introduced in Segformer <cit.> as the backbone of image encoder ω. MiT is a more suitable backbone for image segmentation tasks compared to the vanilla ViT <cit.>. In brief, MiT incorporates overlapped patch embedding to maintain local consistency, spatial-reductive self-attention to accelerate computation, and employs FFN with convolutions to safely replace position embedding. The network of φ to predict the semantic affinity is implemented by an MLP layer. 
The deep network ψ employed to generate the flattened parameters of the mapping function f is constructed by several Conv+ReLU+BatchNorm layers, followed by a linear layer.

Loss.
During training, we use the combined loss functions to optimize overall performance. To improve AIRMF, we utilize the cross-entropy loss ℒ_ce, L1 loss ℒ_1, L2 loss ℒ_2 and gradient loss ℒ_grad

    ℒ_AIRMF = λ_1ℒ_1 + λ_2ℒ_2 + λ_3ℒ_ce +λ_4ℒ_grad,

where λ_1=0.2, λ_2=0.2, λ_3=0.3 and λ_4=0.4 balance the contributions in ℒ_AIRMF. The ablation study to evaluate each component of ℒ_AIRMF is included in Sec. <ref>.



To calculate the affinity loss ℒ_aff, we concentrate on adjacent coordinates during training to leverage their sufficient context while minimizing computational costs. We use a search mask on 𝐌̂ with a predefined radius R as a threshold, efficiently limiting the distance between selected coordinate pairs. This approach ensures that only pixel pairs within the same local search mask are considered, excluding the affinity between distant pixel pairs. The affinity loss term is formulated as follows:


    ℒ_aff   = - 1/|𝒮^-|∑_(i,j) ∈𝒮^-log(1-𝐀𝐟𝐟_pre^i,j) 
        - 1/|𝒮^+|∑_(i,j) ∈𝒮^+log(𝐀𝐟𝐟_pre^i,j),

where 𝒮^+ and 𝒮^- refer to two subsets of positive and negative affinity pixel pairs: 

    𝒮^+ = {(i,j) | (i,j) ∈𝒫, 𝐀𝐟𝐟_pre^i,j = 1},


    𝒮^- = {(i,j) | (i,j) ∈𝒫, 𝐀𝐟𝐟_pre^i,j = 0}.

Our final loss can be written as:

    ℒ= 1/2ℒ_AIRMF + 1/2ℒ_aff.






Training details. During training, we utilize the Adam optimizer <cit.> with an initial learning rate 2.25 × 10^-5 to train our network. The learning rate is decayed to one-tenth at iterations 22,500 and 37,500 in a total of 45,000 iterations. We randomly crop input with a resolution of 224x224 from the original images. During the training, the coarse masks are generated by introducing random perturbations to the ground truth masks with a random IoU threshold ranging from 0.8 to 1.0. The radius R of the local window size when computing the affinity loss is set to 14.

We follow the multi-resolution inference strategy proposed in <cit.>, which can be seen as progressively coarse-to-fine operations. Specifically, the inference begins at a very low resolution around the training image resolution and gradually increases the input’s resolution along the continuous ratio axis Rs ∈ (0.125,0.25,0.5,1). At each refinement stage, the refined mask is continuously concatenated as a coarse mask with input images for the subsequent refinement step.



§ EXPERIMENTS





 §.§ Setups


Datasets. To acquire object information and evaluate our models, we adhere to a dataset setup similar to that of CascadePSP <cit.>. Our training dataset comprises a combination of several individual datasets, totaling 36,572 RGB images accompanied by their corresponding ground truth masks. These images encompass a wide spectrum of over 1,000 categories, drawing from MSRA-10K <cit.>, DUT-OMRON <cit.>, ECSSD <cit.>, and FSS-1000 <cit.>. For testing, CascadePSP introduced a high-resolution image segmentation dataset, named BIG, ranging from 2K to 6K. Additionally, recognizing that the commonly used original PASCAL VOC 2012 dataset lacks pixel-perfect segmentations and designates regions near object boundaries as 'void', <cit.> relabeled 500 objects to create a new dataset suitable for the task of UHR image segmentation.

Metrics. We evaluate our model using Intersection over Union (IoU) and mean Boundary Accuracy (mBA) <cit.>, which robustly computes the segmentation accuracy across different image sizes within a specified radius from the ground truth boundary. Our model is evaluated without resorting to finetuning in various settings, demonstrating the enhancements it achieves.



















 §.§ Quantitative Comparison


 Table <ref> provides a comprehensive comparison of class-agnostic IoU and mBA among our AIRM, SegFix <cit.>, CascadePSP <cit.> and CRM <cit.>. All models are trained on low-resolution images and evaluated on their original high-resolution counterparts. Across both datasets, our method consistently outperforms other methods in terms of IoU and mBA. It's worth noting that SegFix's refinement performances are comparatively lower, as it is not specifically designed for the UHR image segmentation task. In the BIG dataset, our method exhibits an average improvement of around 1.7% for IoU and 1.2% for mBA based on four fundamental segmentation approaches. Similarly, in the relabeled PASCAL VOC 2012 dataset, our method achieves an average increase of approximately 1.1% for IoU and 0.6% for mBA compared to CRM and CascadePSP upon four approaches. These quantitative findings underscore the effectiveness of our model in capturing global semantic features and exploring pixel-wise affinity interrelationships in UHR image segmentation.
 





 §.§ Qualitative Comparison


In big, we present a qualitative comparison of the performance of CascadePSP, CRM, and our AIRM. Remarkably, even without any prior exposure to high-resolution training images, our method consistently produces refined results of exceptional quality across various scales. When employing four distinct input coarse masks, it becomes evident that our approach excels at reconstructing missing parts in comparison to CascadePSP and CRM. For instance, as depicted in big, our method accurately distinguishes the pilot and the passenger seated in the rear of the airplane (the second row) from the segmented object. Additionally, our method precisely delineates the shape of the right rear-view mirror (the third row), achieving a notably high level of precision when compared to the ground truth, which other methods fail to accomplish.


Furthermore, Voc underscores the substantial visual enhancements achieved by our refinement technique on the relabeled PASCAL VOC 2012 dataset, which consists of low-resolution images. A direct comparison highlights the generation of finer details in our outputs, particularly along the boundary region, such as the gap in the back of the chair and intricate feather textures. More results in supplementary material further demonstrate the efficacy of our refinement algorithm.




 §.§ Ablation Study






Affinity Empowered Encoder (AEE). For simplicity, We follow the method in <cit.> to acquire the affinity prediction map in CNN-based backbone. The semantic affinity between a pair of feature vectors is defined in terms of their L1 distance. From Table <ref>, we can find that: 182 Utilizing the transformer-based backbone yields superior segmentation results compared to employing the CNN-based backbone. Specifically, employing Mit-b1 as the image encoder enhances performance by 2.37%/2.17% compared to employing R50. 183 Affinity learning proves beneficial for both CNN-based and transformer-based backbones. Performance improves by 2.89%/1.21% when using R50 and by 2.59%/2.14% when using Mit-b1. 184 Our method, empowered by affinity learning and a transformer-based backbone, achieves the best results.





In addition, we also provide visualized attention maps and learned affinity in affinity. From the visualizations, we observe that the attention map (affinity(b)) only captures coarse-level semantic information, whereas the learned affinity (affinity(c)) provides more reliable semantic information. This outcome strongly validates both our initial motivation and the effectiveness of the AEE module.







Adaptive Implicit Representation Mapping Function (AIRMF). To demonstrate the effectiveness of AIRMF, we replace it with SIRMF and present the comparison results in Table <ref>. We observe that without the proposed AIRMF, the final segmentation results notably decreases across all evaluation metrics, such as the IoU decreases by 1.45 and mBA decreases by 4.00.





In addition, we employ the TSNE <cit.> to reduce the dimension of the predicted adaptive parameters and illustrate that in adaptive. We observe that for input images with a similar semantic appearance, the corresponding mapping parameters are closely related. This demonstrates that the predicted adaptive parameters in AIRMF possess the capability to capture global semantic information.



     

































Computation Cost. We conduct experiments to evaluate the time consumption and the model parameters of each component of our method and the competitors, as shown in Table <ref>. We can find that: 182 
When comparing Transformer and CNN-based backbones, despite an increase in the number of parameters by around 3M (Lines 1 and 3, 2 and 4), our network's exceptional performance remains evident, with the corresponding increase in time being below 1.0%. 183 In comparing AIRMF and SIRMF, the inference time and number of parameters remain nearly identical, as we employ a simple hypernetwork to achieve adaptive effectiveness; however, the results significantly outperform expectations.184 When compared to previous methods such as cascade networks (CasPSP) and conventional implicit functions (CRM), our method significantly enhances precise results in UHR image semantic segmentation by around 4.1% and 4.0%, respectively, while maintaining computation costs well within an acceptable range.

loss.

We conduct an ablation study on the loss function by individually removing each of its components. As shown in Table <ref>, the performance decreases with the exclusion of each component. Moreover, different combinations of the loss only slightly affect training time.

Search mask radius in affinity loss.
In Table <ref>, we also study the segmentation performance on different search mask radius R when generating affinity label for computing affinity loss. Through comparison, we observe that a small value of R may not yield sufficient affinity pairs, whereas a large 
R might compromise the reliability of distant affinity pairs. Thus, we choose R = 14 as our priority.






§ CONCLUSION


 
 
Our study extensively examines how receptive fields impact implicit representation mapping and addresses the limitations of current shared mapping functions. We introduce an innovative framework for Ultra High-Resolution (UHR) image semantic segmentation, comprising the Affinity Empowered Encoder (AEE) and the Adaptive Implicit Representation Mapping Function (AIRMF). This methodological approach facilitates the extraction of pixel-wise features with a much larger receptive field, translating it into a segmentation result enriched with global semantic information. Our meticulous experiments demonstrate that our method significantly outperforms competitors.

Limitations While our proposed method outperforms all competitors, the current approach still adheres to the standard configuration of Ultra High-Resolution Image Segmentation testing, which requires a coarse segmentation mask as input, leading to computational burden. Our forthcoming efforts will focus on resolving this intricate issue, emphasizing the robustness of the framework and enhancing its capability to directly handle high-resolution images.
splncs04



 Paper Title: CC-SAM: SAM with Cross-feature Attention and Context for Ultrasound Image Segmentation.





 









Shreyank N Gowda and David A. Clifton



Department of Engineering Sciences, University of Oxford, OX3 7DQ Oxford, UK Oxford Suzhou Centre for Advanced Research, University of Oxford, Suzhou 215123, Jiangsu, China

CC-SAM: SAM with Cross-feature Attention and Context for Ultrasound Image Segmentation
    Shreyank N Gowda1 David A. Clifton1,2
    
======================================================================================




§ ABSTRACT 
The Segment Anything Model (SAM) has achieved remarkable successes in the realm of natural image segmentation, but its deployment in the medical imaging sphere has encountered challenges. Specifically, the model struggles with medical images that feature low contrast, faint boundaries, intricate morphologies, and small-sized objects. To address these challenges and enhance SAM's performance in the medical domain, we introduce a comprehensive modification. Firstly, we incorporate a frozen Convolutional Neural Network (CNN) branch as an image encoder, which synergizes with SAM's original Vision Transformer (ViT) encoder through a novel variational attention fusion module. This integration bolsters the model's capability to capture local spatial information, which is often paramount in medical imagery. Moreover, to further optimize SAM for medical imaging, we introduce feature and position adapters within the ViT branch, refining the encoder's representations. We see that compared to current prompting strategies to fine-tune SAM for ultrasound medical segmentation, the use of text descriptions that serve as text prompts for SAM helps significantly improve the performance. Leveraging ChatGPT's natural language understanding capabilities, we generate prompts that offer contextual information and guidance to SAM, enabling it to better understand the nuances of ultrasound medical images and improve its segmentation accuracy. Our method, in its entirety, represents a significant stride towards making universal image segmentation models more adaptable and efficient in the medical domain.





§ INTRODUCTION



Deep learning has revolutionized the fields of image <cit.>, video classification <cit.> and medical vision <cit.>, enabling unprecedented accuracy and efficiency in these areas. Medical image segmentation plays a pivotal role in delineating and emphasizing particular organs, tissues, and anomalies within medical scans, forming a cornerstone of computer-aided diagnostic systems <cit.>. The development of numerous deep learning models for autonomous medical image segmentation has signaled a transformative shift in the potential of this technology <cit.>. Nevertheless, these models are typically specialized for distinct tasks and require recalibration when used for different tasks, a limitation which poses significant challenges in medical data.



Segment anything model (SAM) <cit.> has emerged as a versatile foundational model for vision segmentation, gaining significant attention due to its unparalleled ability to segment a wide variety of objects and exhibit powerful zero-shot generalization. With user-driven prompts such as points, bounding boxes, and coarse masks, SAM dynamically segments the relevant objects. This adaptability positions SAM as an ideal tool for a myriad of segmentation use cases, facilitating the amalgamation of several discrete medical image segmentation tasks into a holistic model, enhancing its utility for clinical applications <cit.>.

However, while SAM boasts an extensive dataset (SA1B), its performance diminishes in the medical realm due to the lack of comprehensive clinical annotations <cit.>. To enhance SAM's performance in medical image segmentation, many approaches have been proposed such as tuning the mask decoder <cit.>, applying the LoRA strategy to the image encoder <cit.>, introducing task-specific information using adapters in the ViT image encoder <cit.>. Previous endeavors to adapt SAM for medical image segmentation by optimizing it on medical datasets have encountered limitations, given their tokenization approach that can obfuscate essential local data.

SAMUS <cit.> was subsequently introduced, aiming to transplant SAM's superior segmentation prowess and robust generalization to medical image segmentation, all while ensuring computational efficiency. Notable improvements within SAMUS encompass a redesigned ViT image encoder, integration of a parallel CNN-branch, and the introduction of a cross-branch attention mechanism, among others. 

In this paper, we introduce CC-SAM, a novel model built upon SAMUS. Notably, CC-SAM replaces the adjustable CNN with a fixed one, enhanced with adapters and employs a novel variational attention fusion instead of the cross-attention branch. Utilizing descriptions from Chat-GPT to prompt the model, we achieve a marked improvement in performance over existing prompting strategies tailored for medical-oriented SAM. This new approach significantly improves medical-oriented SAM performance. See Figure <ref> for a comparison with other medical SAM adaptations.

The chief contributions of this paper are delineated as follows:


    
  * The development of CC-SAM: a refined foundation model tailored for universal medical image segmentation, optimized for computational efficiency.
    
  * The introduction of a static CNN with adapters, complementing the ViT encoder of SAMUS and further reducing computational costs.
    
  * The innovative integration of variational attention fusion, enhancing the synergy between CNN and ViT branch features.
    
  * The use of text-based prompts, sourced from Chat-GPT, to articulate the segmentation problem, which significantly amplifies segmentation performance in medical contexts.




§ RELATED WORK







 §.§ Medical Image Segmentation




Medical image segmentation is crucial for identifying and measuring structures in medical images. Before deep learning, methods like thresholding, clustering, and active contour models were common <cit.>. However, CNNs, particularly U-Net <cit.>, have since dominated due to their quality results even with limited data.



Extensions of U-Net, such as DRINet <cit.>, tackled class imbalance and high-resolution outputs. U-Net++<cit.> used nested and dense skip pathways to improve segmentation, while UNeXt<cit.> optimized for speed and efficiency in medical segmentation by using a Convolutional MLP-based network.







The rise of image transformers <cit.> led to new methods. FAT-Net <cit.> introduced feature adaptive transformers for detailed segmentation. UNeTR <cit.> combined transformers with a U-shaped design for 3D medical imaging. Our approach merges CNNs and transformers to harness both local and global feature extraction, creating a hybrid framework that leverages the strengths of both paradigms.



 §.§ Adapting Foundational Models


Foundational models, often pre-trained on large datasets, provide a generic understanding of complex patterns in data. These models are crucial stepping stones in many modern machine learning workflows. However, leveraging them effectively for specific tasks necessitates a thoughtful adaptation strategy. 



Adapter modules <cit.> offer a solution, allowing the main network weights to remain unchanged while adding learnable layers to each Transformer layer. This adds task-specific abilities with less overhead, ensuring efficient transfer learning. This approach keeps the base knowledge while adapting to new tasks.

These modules have excelled in various language tasks <cit.>. For instance, they've achieved top results in question answering with 95% fewer parameters than standard finetuning <cit.>. In textual entailment, AdaMix <cit.> found adapters performed comparably to full finetuning but with only 3% added parameters. They've also been effective in computer vision tasks like image <cit.> and video classification <cit.>. Essentially, they ensure efficient transfer learning across tasks by adding minimal layers to fixed networks.



We build on the idea of adapters and freeze both our CNN and ViT backbones and only use adapters for efficient fine-tuning of both models.



 §.§ Adapting SAM for Medical Images










The Segment Anything Model (SAM)<cit.> has demonstrated notable zero-shot segmentation using various input prompts. However, its performance on medical images is subpar<cit.>.

Several strategies have been proposed to adapt SAM for medical imaging. MedSAM optimizes the mask decoder for efficient fine-tuning. SAMed <cit.> freezes the image encoder and employs a low-rank finetuning strategy while adjusting the prompt encoder and mask decoder. MSA <cit.> uses adapters in the ViT image encoder. To address transformer backbone limitations, SAMUS <cit.> incorporates a CNN-based backbone for improved local feature learning.

Building on SAMUS <cit.>, we optimize its framework. Instead of end-to-end CNN fine-tuning, we utilize a frozen pre-trained CNN and an adapter. We found that using text prompts, especially with Grounding DINO <cit.>, significantly enhances SAMUS's performance. This method is not only more effective but also faster. We use GPT-4 <cit.> to generate text prompts from class labels.





§ PROPOSED METHOD









Figure <ref> provides a detailed view of our method. Essentially, we process an image I using ViT-B <cit.> (pre-trained image encoder of SAM) and ResNet50 <cit.>, the ResNet50 is pre-trained on RadImageNet <cit.>. After feature extraction, adapters refine these representations. Features are then fused using a variational attention mechanism, forming a combined feature for the Mask Decoder of SAM.

For prompts, we use a GPT-4 generated label description and embed it with Med-BERT <cit.>. The Grounding DINO model, untrained on medical data, creates a bounding box from this embedding, which is input to the Prompt Encoder. Together with the variational attention feature, the Prompt Encoder helps the Mask Decoder create the final segmentation mask.



 §.§ Frozen Backbones with Adapters






We've enhanced SAM's image encoder (the ViT branch) for better adaptability to smaller inputs and medical images by introducing a position adapter and five feature adapters, following SAMUS <cit.>. These adapters fine-tune the ViT branch efficiently with fewer parameters.

Specifically, the position adapter modifies positional embeddings to match the embedded sequence resolution. It first downsamples these embeddings using max pooling and then refines them with a convolution operation, enabling the ViT to better manage smaller inputs. Each of the five feature adapters follows a consistent design with a downward linear projection, an activation, and an upward projection. This is mathematically expressed as:


    A(x)=G(xℳ_d)ℳ_u


Here, Here, G denotes the GELU activation function, while ℳ_d∈ℝ^d×d/4 and ℳ_u∈ℝ^d×d/4 are the projection matrices. In this context, 'd' represents the dimension of the feature embedding. 



Unlike SAMUS's end-to-end CNN training, we use a static, RadImageNet <cit.> pre-trained ResNet-50 model. We add a trainable fully-connected layer, acting as an adapter, before its classification layer. The output features from both branches then go into our variational attention fusion block.




 §.§ Variational Attention Fusion Block







A major contribution we've made is the Variational Attention Fusion Block, which adeptly merges local CNN features and global ViT features. While SAMUS introduced a cross-branch attention module, our method, which models uncertainty around these features and applies variational attentional fusion, outperforms it (details in Sec <ref>). See Figure <ref> for an overview.




  §.§.§ Uncertainty Learning







Incorporating uncertainty learning, for an input image I, we derive features s^v from the ViT and s^c from the ResNet-50. Both s^v and s^c∈ℝ^d^h represent a fixed dimension in a latent space. We achieve a consistent size using latent encoders E_v, E_c at each network's end. To address data uncertainty, we design a latent distribution for each sample's content, helping capture semantic relations. This distribution is shaped as a parameterized diagonal Gaussian. This is mathematically represented as:


    p(z^v|s^v)=𝒩(z^v;μ ^v,(σ ^v)^2𝐈)


    p(z^c|s^c)=𝒩(z^c;μ ^c,(σ ^c)^2𝐈)






Here, z^v and z^c are reconstructed vectors. The means, μ^v, μ ^c, capture each mode's core feature, while variances, σ^v, σ^c, indicate the noise-induced uncertainty in predicting these means. A higher variance signifies more uncertainty about the content observed. Both Gaussian parameters depend on the input and are forecasted through MLPs. For instance, for ViT features: μ^v = fθ_v1(sv) and σ^v = fθ_v2(sv), with θ_v1 and θ_v2 being respective model parameters. The same applies to CNN features. Now, each sample's feature representation shifts from deterministic to a stochastic Gaussian-drawn embedding in the latent space.

Due to the non-differentiable nature of sampling, we use a re-parameterization trick <cit.> to maintain gradient flow. We draw random noise η from a normal distribution, independent of model parameters, and produce z^v using it, as detailed in Eq <ref>. We repeat this for z^c.


    z^v=μ ^v+ησ ^v,η∈𝒩(0,1)








  §.§.§ Variational Attentional Fusion






CNNs and ViTs capture different feature aspects from images (local and global). We see these as two modes: v for ViTs and c for CNNs. Combining them isn't straightforward due to different confidence levels. Standard methods, which get weights from the data, miss the unique traits of each mode. To tackle this, we've developed a Variational Attention Fusion (VAF) module. This module aims to capture the nuances between modalities by determining modality-specific weights for a seamless integration.

In short, using features from each mode, a standard attention method creates a probability, a^k (where k is either v or c), as shown in Eq <ref>. Here, a^k shows how much each mode contributes, and W_m and b_m are trainable factors. By having set weights, it picks the right features without being affected by varied confidence levels between modes.



    â^k=W_mz^k+b_m
     a^k=exp(â^k)/∑_k'∈{ v,c}exp(â^k')





Our VAF module uses variational attention weights instead of the point-estimated attention vector â^k. This approach, grounded in a probability distribution (Eq. <ref>), better accounts for uncertainties across modalities.


    â^k ∼ q_θ(â^k | z^k)=𝒩(μ_a^k,(σ_a^k)^2 𝐈), k ∈{v, c}




We used an identity transformation to preserve modality traits in the variational attention. Mean and variance parameters are predicted by MLPs from the input. The (σ_a^k) value indicates confidence levels among modalities. With the VAF method, more confident modality features are enhanced, while less confident ones are reduced, optimizing multimodal data fusion and capturing complementary features. The final representation combines modality-specific outputs, z_c and z_v, using weighted aggregation as shown in Eq <ref>, where W_h is a learnable weight matrix.


    h = ∑_k ∈{ v,c}a^kW_hz^k






 §.§ Guiding the Prompt Encoder




SAM's mask decoder <cit.> needs h and an input from the prompt encoder. While past adaptations of SAM for medical imaging used point prompts, our results show that a reasonably good bounding box enhances performance. Creating this bounding box requires a dedicated model. Using random bounding boxes may hinder results, necessitating specific prompts. We employ Grounding-DINO <cit.> for bounding box generation. Though not exclusively trained for medical images, Grounding-DINO is an effective object detector. We use GPT-4 <cit.> to craft text prompts for specific class labels, and then MedBERT <cit.> generates an embedding input for Grounding-DINO. Whilst this does improve our performance in comparison to using point prompts, we show in supplementary that using the same set of random point prompts, we outperform all other medical foundational models.

Input to GPT-4 "Create a concise description for a medical image analysis task. The task involves using ultrasound images to segment and identify specific anatomical structures or pathologies. Use [target] for to make a generic description that can be adapted for any segmentation target."
Output from GPT-4 "Segment and identify [target] within ultrasound images. This requires the precise delineation of [target] from surrounding tissues, enabling accurate diagnosis and assessment. The challenge encompasses dealing with the inherent variability of ultrasound image quality, including speckle noise, shadowing, and the wide range of anatomical variations among patients. Success in this task is measured by the algorithm's ability to consistently and accurately identify and outline the [target], providing crucial information for medical decision-making processes. " where [target] is modified for each dataset.



 §.§ Loss Function


Our overall loss function can be seen in Eq <ref>. Here, ℒ_BCE and ℒ_D are segmentation losses: binary cross entropy and dice loss respectively. 


    ℒ = ℒ_BCE+ℒ_D+λ _cℒ_c+λ _vℒ_v+λ _aℒ_a




ℒc and ℒv are regularization terms for uncertainty learning, inspired by the variational information bottleneck <cit.>. To grasp uncertainty in a modality, a component is introduced during learning. It ensures the data's distribution, 𝒩(z^v;μ^v,(σ^v)^2), resembles a standard bell curve, 𝒩(ϵ ; 0, 𝐈). This similarity is gauged using Kullback-Leibler divergence (KLD) between the distributions, promoting model diversity and reducing uncertainty, leading to robust features. Eq <ref> shows how ℒv is computed, with analogous methods for ℒc and ℒ_a, replacing z^v with z^c and â^k.


    ℒ_v     =K L(𝒩(z^v ; μ^v,(σ^v)^2) 𝒩(ϵ ; 0, 𝐈)) 
        =-1/2(1+log(σ^v)^2-(μ^v)^2-(σ^v)^2)




§ EXPERIMENTAL ANALYSIS





 §.§ Datasets








To compare with SAMUS <cit.>, we test CC-SAM on the same seven public datasets: TN3K <cit.>, DDTI <cit.>, TG3K <cit.>, BUSI <cit.>, UDIAT <cit.>, CAMUS <cit.>, and HMC-QU <cit.>. We also recreate the US30K dataset from SAMUS for broader comparison. Dataset details are in the supplementary material.

TN3K and TG3K datasets follow TRFE <cit.> for segmentation into training, validation, and testing. BUSI is randomly split in a 7:1:2 ratio for these purposes. CAMUS is initially divided into training and testing per the challenge <cit.>, with 10% of training data later used for validation.

For model generalizability, US30K datasets remain unseen during training and validation. In line with SAMUS, we split US30k into seen and unseen datasets. We then evaluate on unseen datasets (DDTI, UDIAT, HMC-QU) not used in training or validation. This enables direct comparison of generalization with other state-of-the-art models. Moreover, we contrast CC-SAM with other base models by training on the full US30K dataset and testing across different tasks.



 §.§ Implementation Details


We follow SAMUS in setting hyper parameters and adapter sizes. We change the fully tuneable CNN in SAMUS to a static ResNet50. The model is trained using the Adam optimizer with an initial learning rate of 0.01 that reduces by a tenth every 50 epochs for a total of 200 epochs. The encoders are fully connected networks with the latent space having dimensions 1024. All fully connected layers has a Leaky RELU activation function and a dropout probability of 0.5.

We use 1xA100 GPU with 40 GB capacity. But we also verified that a single 3090 Ti with 24 GB memory is enough to train the model. We set a batch size of 32. 



 §.§ Ablation Study



We consider SAMUS <cit.> as our baseline model and ablate our added components. Specifically, we show replacing the end-to-end CNN model with a fixed pre-trained model improves performance. Further adding the FC layer at the end slightly improves performance. Crucially, replacing the cross attention fusion with our variational attention fusion significantly improves performance. Further, replacing the point based prompting with our text-based prompting gives us the best results. We show these results on two of the datasets TN3K and BUSI in Table <ref>.



We do a futher analysis of how the prompting compares with different prompting strategies in the supplementary material.



 §.§ Comparison with state-of-the-art


We consider two scenarios for comparing with state-of-the-art models following SAMUS <cit.>. We compare CC-SAM with task-specific methods and foundational models.



  §.§.§ Task-specific methods


Following SAMUS <cit.>, thirteen state-of-the-art, task-specific techniques are chosen for evaluation, spanning CNN-based, transformer-based, and CNN-Transformer hybrid methods. Methods other than SAMUS include CNN-based techniques encompass U-Net <cit.>, CPFNet <cit.>, CA-Net <cit.>, CE-Net <cit.>, and AAU-Net <cit.>. The transformer-centric methods consist of SwinUnet <cit.>, SETR <cit.>, and MISSFormer <cit.>. The hybrids of CNN and Transformer are TransUNet <cit.>, TransFuse <cit.>, FAT-Net <cit.>, and H2Former <cit.>.







  
Quantitative Analysis.



Table <ref> summarizes the numerical outcomes of various task-specific methods on TN3K, BUSI, CAMUS-LV, CAMUS-MYO, and CAMUS-LA. In comparison, CC-SAM consistently excels in performance across all these five tasks, setting a new benchmark for SOTA results. While CC-SAM surpasses recent SOTA in most benchmarks, it does fall short by a very small margin when evaluated with the Hausdorff distance (HD) on CAMUS-MYO.





  
Qualitative Analysis.



Figure <ref> showcases the segmentation results from various methods, including U-Net <cit.>, AAU-Net <cit.>, MISSFormer <cit.>, H2Former <cit.>, SAMUS <cit.>, and our proposed CC-SAM. Segmenting ultrasound images is notably challenging given their low contrast, inconsistent features, and indistinct object boundaries. Many existing techniques face difficulties in precisely differentiating the target from its background, often resulting in significant false negatives or false positives. In contrast, SAMUS stands out in maintaining the integrity of target areas and minimizing false positives. Our CC-SAM further refines the outcomes observed with SAMUS, a testament to the intrinsic strengths of SAM and the tailored modifications and innovations integrated into CC-SAM.







  
Generalization Ability.



Evaluating the generalization performance of various task-specific methods is crucial as it provides insights into how well these methods can adapt and function across unfamiliar or unseen datasets. Figure <ref> offers a quantitative assessment of this aspect. When we assess performance discrepancies between known (seeable) and unknown (unseen) datasets, CC-SAM stands out by displaying the most minimal performance drop among all methods across the three segmentation tasks. This underlines the remarkable generalization prowess of CC-SAM, emphasizing its resilience and versatility in a variety of medical image segmentation situations.





  §.§.§ Foundational models


We also compare with five state-of-the-art foundational models. These include the original SAM <cit.>, SAMed <cit.>, SAMUS <cit.>, MSA <cit.> and MedSAM <cit.>. SAM is  trained on SA-1B and kept frozen due to the cost involved in fine-tuning. All other foundational models are pre-trained on the US30k dataset and evaluated using Dice and HD across TN3K, BUSI, CAMUS-LV, CAMUS-MYO,
and CAMUS-LA datasets.





  
Quantitative Analysis.




Table <ref> summarizes the numerical outcomes of various foundational methods on TN3K, BUSI, CAMUS-LV, CAMUS-MYO, and CAMUS-LA. In comparison, CC-SAM consistently excels in performance across all tasks, setting a new benchmark for foundational models.



  
Qualitative Analysis.




Figure <ref> shows the qualitative segmentation results from various foundational models, including SAM, MedSAM, SAMed, MSA, SAMUS, and CC-SAM. SAM, without adjustments for medical images, fails at segmentation. However, MedSAM, SAMed, and MSA, through tuning, recover some segmentation ability but struggle with accurately defining borders, resulting in many false positives and negatives. In contrast, SAMUS excels in identifying precise segmentation borders, even in low contrast, suggesting the benefit of integrating local data with the image encoder for better boundary and shape retention in medical image segmentation. CC-SAM goes further, improving upon SAMUS's performance and setting new benchmarks.







  
Generalization Ability.


Generally, foundation models trained on US30K exhibit a much improved generalization performance in medical image segmentation tasks compared to the original SAM. When considering the three specific segmentation tasks: thyroid nodule segmentation, breast cancer segmentation, and myocardium segmentation, it's clear that all foundation models face significant challenges with myocardium segmentation, while they perform relatively well in the breast cancer segmentation task. CC-SAM outperforms all foundational models significantly. This highlights the remarkable generalization prowess of CC-SAM, as it consistently surpasses other foundation models, especially in unfamiliar domains. Figure <ref> offers a quantitative assessment of this aspect. We see that the original SAM performs the worst, which is expected and the proposed CC-SAM obtains a new state-of-the-art on all datasets.






  
Deployment Cost Comparison with Other Foundational Models

We compare CC-SAM with other foundational models in a more detailed manner. We look at `Generalization Ability/Dice (%)', `GPU Memory/G', `Parameters/M', `Computation/GFLOPs', `Inference Speed/FPS' and `Segmentation Performance/Dice (%)'.The spider plot of this can be seen in Figure <ref>.







§ CONCLUSION


The Segment Anything Model (SAM), while proficient in natural image segmentation, faced hurdles in medical imaging due to issues with low contrast, faint boundaries, and other complexities. 
In this study, we identified and addressed the limitations of the Segment Anything Model (SAM) when applied to medical imaging, especially with images that have inherent challenges like low contrast and intricate morphologies. Our enhanced approach integrates a CNN branch with SAM's original Vision Transformer (ViT) encoder, utilizing a unique variational attention fusion module to capture vital local spatial information present in medical images. By introducing feature and position adapters within the ViT branch, the model's representations are further refined. Notably, we found that using text prompts, especially those generated leveraging ChatGPT's capabilities, significantly boosts SAM's performance in ultrasound medical segmentation. These innovations mark a pivotal step towards improving the adaptability and efficiency of universal image segmentation models in the medical field.



  
Acknowledgements

DAC was supported by the Pandemic Sciences Institute at the University of Oxford; the National Institute for Health Research (NIHR) Oxford Biomedical Research Centre (BRC); an NIHR Research Professorship; a Royal Academy of Engineering Research Chair; the Wellcome Trust funded VITAL project (grant 204904/Z/16/Z); the EPSRC (grant EP/W031744/1); and the InnoHK Hong Kong Centre for Cerebro-cardiovascular Engineering (COCHE).






splncs04



 Paper Title: ESIQA: Perceptual Quality Assessment of Vision-Pro-based Egocentric Spatial Images.


On Approximating Volumes of Clipped Hypercubes
    Costandin Marius
    August 5, 2024
==============================================

§ ABSTRACT 









With the development of eXtended Reality (XR), head-mounted shooting and display technology have experienced significant advancement and gained considerable attention.
Egocentric spatial images and videos are emerging as a compelling form of stereoscopic XR content. 
Different from traditional 2D images, egocentric spatial images present challenges for perceptual quality assessment due to their special shooting, processing methods, and stereoscopic characteristics. However, the corresponding image quality assessment (IQA) research for egocentric spatial images is still lacking.
In this paper, we establish the Egocentric Spatial Images Quality Assessment Database (ESIQAD), the first IQA database dedicated for egocentric spatial images as far as we know. Our ESIQAD includes 500 egocentric spatial images, containing 400 images captured with the Apple Vision Pro and 100 images generated via an iPhone’s “Spatial Camera" app. The corresponding mean opinion scores (MOSs) are collected under three viewing modes, including 2D display, 3D-window display, and 3D-immersive display. Furthermore, based on our database, we conduct a benchmark experiment and evaluate the performance of 22 state-of-the-art IQA models under three different viewing modes.
We hope this research can facilitate future IQA research on egocentric spatial images. The database is available at https://github.com/IntMeGroup/ESIQA.




    


§ INTRODUCTION




Egocentric shooting and displaying technology has gained considerable attention recently, which can provide convenient capturing methods and immersive experiences through head-mounted displays (HMDs) <cit.>, such as Apple Vision Pro, Meta Quest 3, and Google Glass, etc. 
Egocentric spatial images, captured from the first-person perspective, are emerging as an important form of stereoscopic content.
As shown in Figure <ref>, egocentric spatial images are captured with HMDs in real-world scenes and can be transmitted and viewed on various terminal devices. 
For binocular displays, since a spatial image is composed of a pair of disparity views to simulate human binocular vision, it is possible to directly create a three-dimensional (3D) effect and provide an immersive experience.
For conventional flat displays, generally only the left view, i.e., a standard two-dimensional (2D) image, is presented.











Egocentric spatial images, taken by a wide variety of individuals from professional photographers to amateurs, exhibit significant differences in visual quality <cit.>. 

This quality variation is further compounded by the dynamic and uncontrolled environments when capturing egocentric spatial images, leading to common distortions such as under/over exposure, low visibility, noise, color shift, etc <cit.>.

Moreover, since the captured spatial images can be viewed on any device, the corresponding quality of experience (QoE) may also be different. Therefore, it is important to develop image quality assessment (IQA) metrics of egocentric spatial images to better maintain and optimize QoE on various devices.

















Over the past few decades, many studies have focused on the traditional 2D image quality assessment problem, with the establishment of several IQA databases, e.g., TID2013 <cit.>, KonIQ-10K <cit.>, and KADID-10k <cit.>, and the development of many advanced IQA models <cit.>.
However, traditional IQA models are inadequate for assessing the quality of egocentric spatial images due to their unique immersive experiences, which have a distinct impact on QoE. Moreover, spatial images consist of two distinct views (left and right), which precludes the direct application of traditional IQA models.

With the diversification of image forms, some studies have focused on 3D IQA (also known as stereoscopic IQA) <cit.> and VR IQA <cit.>. However, existing 3D IQA metrics fail to consider the immersive experience of egocentric spatial images, and VR IQA metrics cannot effectively evaluate their stereoscopic effects. These metrics are unsuitable for assessing the quality of egocentric spatial images.

As far as we know, there is a lack of IQA studies for egocentric spatial images, which have significant differences compared with traditional 2D images, 3D images, and VR images in terms of formats, characteristics, and applications. Therefore, there is an urgent need for dedicated research on egocentric spatial image quality assessment.


To better understand human visual preferences for egocentric spatial images and facilitate the development of IQA models tailored to these images, we construct the first Egocentric Spatial Images Quality Assessment Database, termed ESIQAD. The database contains 500 egocentric spatial images with diverse scenes and the corresponding collected human perceptual quality ratings on three viewing modes, including 2D, 3D-window, and 3D-immersive. Specifically, 400 spatial images are captured using the Apple Vision Pro, and 100 spatial images are generated by the images taken by the iPhone’s “Spatial Camera" application. Based on the collected egocentric spatial images, we conducted subjective experiments to evaluate the QoE of these images, in which 22 participants were included to provide their perceptual quality ratings under three viewing modes. Based on our ESIQAD, we conduct a benchmark experiment to evaluate the performance of 22 state-of-the-art IQA models under three viewing modes. Our contributions are summarized as follows:


    
  * We establish a large-scale quality assessment database for egocentric images, named ESIQAD, which is the first IQA database for egocentric spatial images to the best of our knowledge.
    
  * We analyze the human preference characteristics for egocentric spatial images under three viewing modes based on our ESIQAD.
    
  * We conduct a benchmark experiment by evaluating the performance of numerous state-of-the-art IQA models based on our ESIQAD.































































































































































































































§ DATABASE CONSTRUCTION



To facilitate the egocentric spatial IQA research, we construct a large-scale database, termed ESIQAD, and conduct the corresponding subjective experiment to derive the subjective quality scores in terms of three viewing modes.






 §.§ Content Collection




Firstly, we utilized the Apple Vision Pro and the iPhone 12 as the shooting devices to collect spatial images from a variety of scenes, including both indoor and outdoor environments, different weather conditions, and various times of day. These images feature various  characteristics including brightness, contrast, colorfulness, spatial details, etc., and have different degradations including noise, blur, etc. The raw images were taken by ten photographers, representing a wide range of user perspectives and preferences. 


Specifically, we collected 400 spatial images captured with the Apple Vision Pro, and 100 spatial images generated from the images taken by the iPhone, referred to as captured images and generated images, respectively. For the 100 generated images, we used the “Spatial Camera” app to automatically take a 2D image as the left view and then generate the right view to form a spatial image. 
It should be noted that the 100 generated images have their corresponding images with the same scenes in the 400 images captured by Apple Vision Pro to facilitate the study of human visual preferences.
The captured images have a fixed resolution of 2560 × 2560 per view, while the generated images have a resolution of 4032 × 3024 per view. Figure <ref> demonstrates sample images from our ESIQAD, where all samples are illustrated in the main view (left view). 



 §.§ Subjective Experiment




  §.§.§ Experiment setup



Based on the collected images, we then conducted a subjective experiment to gather quality scores of human perception on egocentric spatial images across different viewing modes, including 2D, 3D-window, and 3D-immersive, as shown in Figure <ref>. 

For the 2D viewing mode, egocentric spatial images were displayed on a 23.8-inch Redmi T238NF monitor with the resolution of 1920 × 1080, driven by a Mac Mini M2 PC. We developed a quality scoring interface using Python Tkinter, which was used to present egocentric spatial images (only the left view) and collect subjective quality ratings. The interface allows users to freely browse the previous and next images, and record the quality scores through a scroll bar.

For the 3D-window and 3D-immersive viewing modes, we utilized the Apple Vision Pro device as the head-mounted display, which features the resolution of 3660 × 3200 per eye, the 120 horizontal field of view (FoV), and the 100Hz refresh rate. For the 3D-window mode, egocentric spatial images appeared as small windows within the HMD, allowing users to maintain awareness of the surroundings. For the 3D-immersive mode, egocentric spatial images were shown in a wider immersive FoV, which provides an immersive egocentric experience. In both 3D viewing modes, all images were displayed through the built-in application “Photos” and displayed at their native resolutions to prevent the scaling distortion. Across all three viewing modes, egocentric spatial images were presented in a random order. 



  §.§.§ Experiment methodology

We adopted the single stimulus absolute category rating (SSACR) method to collect subjective quality ratings for egocentric spatial images under three viewing modes, with the rating scale ranging from 1 to 10. 

Under the 2D viewing mode, the viewers were seated at a distance of about 2 feet from the monitor in a laboratory environment with normal indoor illumination, and utilized the scoring interface to assign quality scores. 

Under the two 3D viewing modes, participants were instructed to wear the Apple Vision Pro and select either the 3D-window or 3D-immersive viewing mode via the “Photos" app. A keyboard was placed
in front of the participants to enter the scores. Participants manually entered their scores after viewing each image and proceeded to the next image using gestures.




A total of 22 graduate students (11 males and 11 females) were invited to participate in the subjective experiment. Before the experiment, all participants completed a standard procedure in visual testing, i.e., the Snellen visual acuity test <cit.>. All participants were confirmed normal or corrected-to-normal vision. 
For all three viewing modes, the participants were instructed to assess the egocentric spatial images based on both distortion and aesthetic quality, providing ratings with an overall quality score. The whole experiment is split into 2 sessions, each of which includes a subjective quality evaluation process for 750 samples (250 spatial images × 3 modes). All participants completed both sessions, producing a total of 33,000 quality ratings (22 participants × 1,500 ratings). After the scoring sessions, each participant was asked to answer a questionnaire regarding their experience. Details of the questionnaire are given in Section <ref>.





 §.§ Subjective Data Processing and Analysis



  §.§.§ MOS Calculation and Analysis
 
We perform the same processing methods for the collected scores under three viewing modes respectively. Firstly, we follow the subjective data processing guidelines recommended by ITU <cit.> for outlier detection and subject rejection. 
For 2D and 3D-immersive viewing modes, 1 of the 22 participants was deemed as an outlier and excluded, respectively. For the 3D-window viewing mode, none of the 22 participants were excluded.
Then, we convert the raw scores from the subjects into normalized Z-scores, which range between 0 and 100. Subsequently, we calculate the average of these Z-scores to derive the mean opinion scores (MOSs), which are formulated as follows: 


    z_i j  =r_i j-μ_i/σ_i,    z_i j^'=100(z_i j+3)/6,


    MOS_j  =1/N∑_i=1^N z_i j^',

where r_ij is the original score of the i-th subject on the j-th image, μ_i and σ_i represent the mean rating and the standard deviation given by subject i, respectively, and N is the total number of subjects. 




In the following analysis, we utilize original MOSs, ranging from 1 to 10, instead of normalized scores to facilitate clearer comparisons across different viewing modes.
Figure <ref> (a) presents the histograms of the MOS distribution over the three viewing modes, indicating that the perceptual quality scores are widely distributed in the [1,10] interval in each mode. 
In addition, Figure <ref> (b) and (c) demonstrate the MOS distributions of 400 captured images and 100 generated images, respectively. It can be observed that the MOS distribution for captured images centers around 5 in all three modes, while it centers above 5 for generated images. 

This is mainly due to the fact that the color of the images captured by the iPhone is generally enhanced to be more comfortable for human vision perception, while the color of the images captured by Apple Vision Pro is generally kept real.
Moreover, for captured images, the perceptual quality is higher in 3D-immersive and 3D-window viewing modes compared to 2D viewing mode. However, for generated images, the perceptual quality is better in 2D viewing mode than in 3D viewing modes.

The results shown in Figure <ref> (a) indicate that the MOS fitting curves for the two 3D viewing modes are located at the right of the 2D viewing mode, manifesting higher perceived image quality in the 3D environments. 
To facilitate a more direct comparison of the perceptual quality of the same image across different viewing modes, we calculate the MOS difference for each image between the three modes. The differences are illustrated in the frequency distribution histograms as shown in Figure <ref>.

The results indicate that the perceptual quality of most images in 3D-window and 3D-immersive scenes is comparable but generally superior to that observed on 2D displays.



To compare the perceptual quality of captured and generated images with identical scenes, we compute the MOS difference between 100 matched pairs of captured and generated images across three viewing modes by subtracting the MOS of generated images and their corresponding captured images. The distribution histogram of the differences is depicted in Figure <ref>. 

The results indicate that the perceptual quality of most generated images exceeds that of their corresponding captured images. This improvement mainly results from image enhancement methods applied during the image generation.










  §.§.§ Questionnaire Analysis

Post-test questionnaires can help better understand the QoE of egocentric spatial images and design future studies. All questions are shown in Figure <ref>. For Q1 and Q2, we derive ranking scores for the three viewing modes based on the sorting results, which reflect the overall ranking of the options.

The ranking score for a viewing mode is calculated by 

    S = ∑ (f × w)/n,

where S indicates the ranking score for this viewing mode, f represents the frequency of each rank, w denotes the weight assigned to each rank (e.g., the top rank is weighted 3, and the lowest rank is weighted 1), and n is the total number of participants. 
For Q3 and Q4, we compute the average scores from questionnaire responses to evaluate user dizziness.
The statistical results of the replies and some general observations are as follows.

    
  * Q1: 3D-immersive mode, 3D-window mode, and 2D mode obtain ranking scores of 2.27, 2.18, and 1.55, respectively. The results suggest that most subjects prefer the experience of 3D-immersive and 3D-window modes viewing in Apple Vision Pro compared to the experience of the 2D mode viewing on the screen display, which is consistent with our previous analysis in Section <ref>.
    
  * Q2: We derive the ranking scores for each viewing mode: 3.00, 1.73, and 1.14 for 3D-immersive, 3D-window, and 2D modes, respectively. The results indicate participants generally consider that the 3D-immersive scene offers the most immersive experience.
    
  * Q3 & Q4:  The scores for dizziness in the 3D-immersive and 3D-window viewing modes are 4.27 and 2.27, respectively. This result suggests that while the 3D-immersive mode offers a superior immersive experience, it also induces more significant dizziness compared to the 3D-window mode.






  §.§.§ Database Attributes Analysis

Following the methodologies outlined in <cit.> and <cit.>, we conduct statistical analysis for our ESIQAD in terms of five low-level vision feature dimensions, including sharpness, brightness, colorfulness, contrast, and cumulative probability of blur detection metric (CPBD). Considering that the ranges of these features are different, we normalized their values to [0, 1] for better illustration.
Figure <ref> shows the kernel distribution of each feature for our ESIQAD. It can be observed that the spatial images in our database have a wide distribution in brightness, colorfulness, contrast, and CPBD features, showing their extensive diversity. However, our database shows a narrow distribution range for the sharpness feature, notably around 0.3. 

















































































































































§ EXPERIMENT





 §.§ Benchmark Models


To evaluate the performance of various existing IQA models on predicting human visual perception for egocentric spatial images, we employ 22 state-of-the-art no-reference (NR) IQA models for comparison, which are designed for traditional 2D images. The selected models can be categorized into two groups:



  ∙Hand-crafted models, which include BRISQUE <cit.>, CORNIA <cit.>, QAC <cit.>, BMPRI <cit.>, NIQE <cit.>, ILNIQE <cit.>, HOSA <cit.>, FISBLIM <cit.>, LPSI <cit.>, BPRI-PSS <cit.>, BPRI-LSSs <cit.>, BPRI-LSSn <cit.>, and BPRI <cit.>.


  ∙Deep learning-based models, including CNNIQA <cit.>, Resnet18 <cit.>, Resnet34 <cit.>, VGG16 <cit.>, VGG19 <cit.>,  HyperIQA <cit.>, WaDIQaM <cit.>, MANIQA <cit.>, and TReS <cit.>.


For traditional hand-crafted IQA models, we apply these models to directly predict the quality scores of the left and right views of an egocentric spatial image, respectively. The predicted score from the left view, which is displayed in the 2D viewing mode, directly serves as the 2D quality score for the image. 

For both 3D-window and 3D-immersive modes, the quality score for the egocentric spatial image is calculated by averaging the scores from the left and right views.

For deep learning-based IQA models, we train and test these models on our ESIQAD. We partition the database into training and testing sets with the ratio of 4:1. The training parameters are set the same as those in the officially released version. 
Considering that these deep learning-based IQA models are designed for traditional 2D images, we feed the left view of spatial images into the models to predict quality scores for the 2D viewing mode. 
Since spatial images cannot be directly input into these models, we fine-tune the architectures of these deep learning-based models to accommodate a dual-image (6 channels) input, comprising both left and right views of the spatial image. These fine-tuned models are then utilized to predict quality scores under 3D-window and 3D-immersive viewing modes.



 §.§ Performance Analysis



We evaluate the performance of these benchmark models using three evaluation criteria, i.e., Spearman rank correlation coefficient (SRCC), Pearson linear correlation coefficient (PLCC), and Kendall’s rank correlation coefficient (KRCC). These three metrics are utilized to measure the prediction monotonicity.
Before computing these criteria, predicted scores are normalized using a five-parameter logistic function <cit.>:

    ŷ=β_1(0.5-1/1+e^β_2(y-β_3))+β_4 y+β_5,

where {β_i | i=1,2, …, 5} are parameters to be fitted, y and ŷ indicate the predicted scores and the corresponding mapped scores, respectively. 

Table <ref> shows the performance of the aforementioned 22 state-of-the-art models. It can be observed that most hand-crafted models show poor performance in evaluating the perceptual quality of egocentric spatial images in 2D viewing mode, and even perform worse for the evaluation in 3D-window and 3D-immersive modes. Compared to the hand-crafted models, deep learning-based models generally exhibit better performance across all three modes. Among all the IQA models, MAINIQA <cit.> achieves the best performance across three modes, demonstrating excellent prediction ability and strong generalization capabilities in different modes.
Across all models, the performance for 2D-viewing-mode quality assessment is generally better than that for the 3D-window and 3D-immersive settings. The result stems from the fact that these metrics are initially designed for 2D IQA and fail to consider the stereoscopic and immersive effects inherent in 3D scenes, which significantly influence the perceptual quality. 
Additionally, most models exhibit slightly improved performance in predicting quality in 3D-window modes compared to 3D-immersive modes, which manifests that 3D-immersive mode presents greater challenges in predicting its QoE.

In future work, we will utilize the left and right views and their disparities in spatial images to build a quality assessment model with better prediction performance and suitable for different viewing modes.















§ CONCLUSION

In this work, we establish the first egocentric spatial images database, termed ESIQAD, comprising 400 captured images and 100 generated images along with their MOSs under three viewing modes, i.e., 2D, 3D-window, and 3D-immersive modes. Based on our ESIQAD, we analyze the human visual perception of these images across various viewing modes and conduct a benchmark study by evaluating the performance of 22 state-of-the-art NR IQA models. The results from our benchmark study indicate that most current advanced IQA models fail to accurately assess the perceptual quality of spatial images, particularly under two 3D settings. 
It is worthwhile to explore the stereoscopic features of spatial images and effectively utilize disparity information to further improve the performance of egocentric spatial image quality assessment metrics in the future.

    ieeenat_fullname
    







 Paper Title: Deep Learning in Medical Image Classification from MRI-based Brain Tumor Images.







Deep Learning in Medical Image Classification from MRI-based Brain Tumor Images



    Xiaoyi Liu
Ira A. Fulton Schools of Engineering 

Arizona State University

Tempe, USA 

xliu472@asu.edu
Zhuoyue Wang
Department of Electrical Engineering and Computer Sciences 

University of California, Berkeley

Berkeley, USA 

zhuoyue_wang@berkeley.edu

    August 5, 2024
=====================================================================================================================================================================================================================================================================






§ ABSTRACT 
Brain tumors are among the deadliest diseases in the world. Magnetic Resonance Imaging (MRI) is one of the most effective ways to detect brain tumors. Accurate detection of brain tumors based on MRI scans is critical, as it can potentially save many lives and facilitate better decision-making at the early stages of the disease. Within our paper, four different types of MRI-based images have been collected from the database: glioma tumor, no tumor, pituitary tumor, and meningioma tumor. Our study focuses on making predictions for brain tumor classification. Five models, including four pre-trained models (MobileNet, EfficientNet-B0, ResNet-18, and VGG16) and one new model, MobileNet-BT, have been proposed for this study. 



brain tumor, deep learning, computer vision, medical image classification, transfer learning




§ INTRODUCTION

Brain tumors are diseases that are difficult to cure and have a high mortality rate. Magnetic Resonance Imaging (MRI) is one of the tests needed for diagnosing brain tumors. Accurate and timely detection through MRI scans is essential for enhancing patient outcomes. Unlike Alzheimer’s disease, which uses intelligent interaction therapy, most brain tumors need surgery <cit.>. In our research, we aim to utilize the brain tumor MRI dataset to classify four types of brain tumors: glioma, meningioma, pituitary tumors, and the absence of tumors. Within our paper, pre-trained models, including MobileNetV2, ResNet-18, EfficientNet-B0, and VGG16, were analyzed for the brain tumor classification task. A new model, MobileNet-BT, based on MobileNetV2, was proposed.




§ BACKGROUND




 §.§ Deep Learning


Machine learning is heavily used nowadays, with strategies such as random forest and XGBoost <cit.>. Deep learning, outperformed many machine learning techniques, is widely used across different industries <cit.>. Many domains, including speech recognition, image classification, and object detection, have greatly improved by using deep learning technologies, achieving cutting-edge results in various applications, including brain tumor detection and segmentation<cit.>. Natural Language Processing (NLP), computer vision (CV), and many other areas benefit from deep learning technologies. For natural language processing, applications such as automatic news generation <cit.>, real estate transactions <cit.>, content recommendation systems <cit.>, financial risk detection <cit.>, psychiatric behavior understanding <cit.>, and ChatGPT are good examples of NLP. The large language model can be further improved by using transformers <cit.>. Recurrent neural network (RNN) is widely used in NLP <cit.>. For computer vision, CNN is the widely used architecture. Other areas in deep learning that could be beneficial such as in the software development process <cit.>, facial expression recognition, and credit risk management for banks <cit.>. Even though deep learning is growing dramatically, security is worth concern since there are many Backdoor/Trojan attacks <cit.> <cit.>.

In this paper, CNN models are analyzed for brain tumor classification tasks. More specifically, pre-trained models MobileNetV2, EfficientNet-B0, ResNet-18, and VGG16 will be studied utilizing the brain tumor dataset. Additionally, a new model, MobileNet-BT, based on MobileNetV2, is introduced, which achieves higher accuracy and F1-score.



 §.§ CNN


Convolutional Neural Networks (CNN) are extensively employed in the domain of CV. The main components of CNNs are convolutional layers, pooling mechanisms, activation functions, and fully connected networks. The convolutional layers are fundamental to the CNN architecture. Learnable filters, known as kernels, are applied to images, sliding through them to detect specific features such as edges, textures, or patterns <cit.>. Pooling layers decrease the spatial resolution of the images. The downsampling of feature maps enables the model to concentrate on the most important aspects while reducing the risk of overfitting <cit.>. Non-linear activation functions enable the model to learn complex patterns. Commonly used activation functions like ReLU help mitigate the vanishing gradient problem <cit.>. Fully connected layers make the final predictions. Thanks to technologies like vision model sparsification, the more complex model could be trained in CNN  <cit.>. CNNs are utilized in numerous CV tasks, such as image classification, object recognition, and semantic segmentation, among others. <cit.>. Many computer vision technologies can be applied to medical images. Medical image reconstruction and enhancement can help reduce noise, artifacts, and other problems <cit.>, and models like Lightweight Information Split Network can reconstruct a super-resolution model <cit.>. Additionally, nodule recognition and detection can assist in identifying abnormalities at an early stage, which could save many lives <cit.>. 



 §.§ Pituitary Tumor

Pituitary tumors are unusual masses that form in the pituitary gland. They are very common, but fortunately, most are invariably benign, which means they are not cancer. Pituitary tumors may hypersecrete hormones, which include but are not limited to prolactin, growth hormone, and adrenocorticotropic hormone<cit.>. The pituitary tumor MRI-based image can be found in Fig.1 below.



 §.§ Meningioma Tumor


Meningiomas are the most common primary intracranial tumor. Although often perceived as entirely benign, meningiomas are frequently linked to significant health issues, including localized neurological impairments, seizures, and reduced quality of life<cit.>. The meningioma tumor MRI-based image can be found in Fig.1 below.



 §.§ Glioma Tumor


Gliomas are the most prevalent type of brain tumor. Gliomas are classified into four grades, ranging from grade I, which is benign (noncancerous), to grade IV, which is the most aggressive. Grade IV gliomas are the most aggressive and frequently occurring gliomas <cit.>. The MRI-based image of the glioma tumor can be found in Fig.1 below.





 §.§ Pre-trained Models




  §.§.§ MobileNetV2



MobileNets are designed to work efficiently, balancing efficiency and accuracy. The model is designed to work in mobile or other resource-constrained environments, reducing the required computational power while maintaining high accuracy <cit.>. For MobileNetV2, inverted residuals, linear bottlenecks, and ReLU activation functions are introduced, achieving better accuracy compared to MobileNetV1 while maintaining a relatively small model size<cit.>.



  §.§.§ EfficientNet-B0



EfficientNet models have different versions, starting with EfficientNet-B0 and going up to B7. These versions increase in size and complexity using a new scaling technique called compound scaling. This method for EfficientNet can be described by the following formulas:


    Depth:      d = α^ϕ
    Width:      w = β^ϕ
    Resolution:      r = γ^ϕ


Where:

    
  * α, β, and γ are the scaling coefficients for depth, width, and resolution, respectively.
    
  * ϕ is a user-defined parameter that determines the overall scaling factor.


Using compound scaling can dramatically increase performance when compared to other scaling techniques such as width scaling, depth scaling, and resolution scaling <cit.>.





  §.§.§ ResNet-18


ResNet-18, introduced by Kaiming He and colleagues in 2016, is a convolutional neural network architecture. The residual blocks are the main innovation addressing vanishing/exploding gradient problems. It allows the CNN model to go deeper<cit.>. The basic residual block used in ResNet-18 can be described by the following equation:


    y = ℱ(x, {W_i}) + x


Where:


    
  * x represents the input to the residual block.
    
  * ℱ(x, {W_i}) represents the stacked convolutional layers, batch normalization, and ReLU activations.
    
  * y represents the output of the residual block.




  §.§.§ VGG16


The final convolutional neural network architecture examined in this paper is VGG16. It was first proposed by Visual Geometry Group (VGG) in 2014. The architecture includes 13 convolutional layers and 3 dense layers. It achieved great performance on the ImageNet dataset at that time. The downside of VGG is having 138 million parameters, which could result in slow training time <cit.>.







§ METHODOLOGY




 §.§ Dataset

The Brain Tumor MRI Dataset is a publicly available dataset used in this research paper <cit.>. It comprises 7023 images, with 2000 images without tumors, 1757 pituitary tumor images, 1621 glioma tumor images, and 1645 meningioma tumor images. The dataset is subsequently split into 0.8 for training, 0.1 for validation, and 0.1 for testing. In this research, supervised learning is used since all the labels are assigned to each image. When data is unlabeled, self-supervised learning strategies can be applied <cit.>.



 §.§ Image Augmentation and Preprocessing

For training datasets, image augmentation and preprocessing are used to avoid overfitting, especially when the dataset has a relatively small number of training images. Images are randomly flipped along both horizontal and vertical axes with a 50% chance, the images are then randomly rotated by ±10 degrees. After that, all images are resized into 224 by 224 pixels. This step is necessary for batch processing in the CNN models. Since all images from datasets are in squares, there is no additional need for cropping the images. Then, all images are converted into tensors with the range of [0, 1]. Finally, normalization is applied to the training images.



 §.§ Learning Rate and Epoch

The learning rate started with 0.001 when training. Then, a learning rate scheduler was applied. The learning rate will be multiplied by 0.1 every 8 epochs. It will help the model to converge properly. The number of epochs is determined by the validation dataset's accuracy. If the validation accuracy does not improve for 8 successive epochs, the training stops. This will make sure no overfitting was done for the training model.



 §.§ Pretrained Models


Four pre-trained models were tested with the dataset: MobileNetV2, ResNet-18, EfficientNet-B0, and VGG16 achieved accuracies of 0.8445, 0.8659, 0.8933, and 0.9497, respectively, with the number of parameters being 3.5 million, 11.7 million, 5.3 million, and 138 million, respectively. There are techniques such as Learning from Teaching (LoT) that could enhance generalization and potentially achieve better outcomes <cit.>. 

All four models were implemented using transfer learning, with all layers except the last one frozen. The accuracy and F1-score of the test dataset are good but could be better with additional operations. 



 §.§ Proposed Deep Learning Models


Pre-trained models are typically trained on extensive datasets like ImageNet. The size of ImageNet and the dataset we use in this research differ significantly. Medical image classification is far different from generic image classification. By unfreezing more layers from the pre-trained model, we can potentially make better predictions for specific datasets and tasks. After comparing all four models, we observed that although MobileNetV2's accuracy is slightly lower compared to the other models, with only 3.5 million parameters and an accuracy of 0.8445, the training time and efficiency of using MobileNetV2 are very competitive when compared with VGG16, which uses 138 million parameters to achieve an accuracy of 0.9497.

Based on the analysis above, the MobileNet-BT model was used in the brain tumor classification task. MobileNet-BT utilized the pre-trained weights of MobileNetV2. After that, all the layers of MobileNetV2 were unfrozen. This step enables the model to capture more specific characteristics from the brain tumor dataset. Then, the final layer was replaced by a custom classifier. The fully connected layer initially incorporates a dropout layer with a probability of 20%, which aids in avoiding overfitting. It is then followed by a linear layer with 1000 output features. An additional dropout layer with a probability of 20% was included. Lastly, a linear layer with four output classes was added. The customized classifier allows the model to learn different levels of abstraction. The dropout layers, which randomly deactivate some neurons, help prevent the model from relying too heavily on certain neurons. The additional dense layer enables the model to capture more complex patterns from the dataset.



 §.§ Performance Measure


When measuring the performance of different models on the test set, we used accuracy,  F1-score, precision, recall, and average loss as the metrics. All of these are good measurements for classification tasks. TP (True Positive), FP (False Positive), FN (False Negative), and TN (True Negative) are used in the equations below. All the equations can be found as follows:


    Accuracy = TN + TP/TP + FP + TN + FN



    Recall = TP/FN + TP



    Precision = TP/TP + FP



    F1-Score = 2 ·Precision·Recall/Recall + Precision


For average loss, cross-entropy was used to calculate average loss, where N is the number of classes, y_i is the true label, and ŷ_i is the predicted probability for class i. The equation is given below:


    L = -∑_i=1^N y_i log(ŷ_i)


Cross-entropy was used as the criterion for all tests conducted in this paper due to its wide use in classification tasks. It effectively manages the discrepancy between the predicted probabilities and the actual outcomes. <cit.>. 















§ EVALUATION AND DISCUSSION OF RESULTS


Different metrics are applied when comparing the performance of different models for the brain tumor classification task. For pre-trained models, MobileNetV2, ResNet-18, EfficientNet-B0, and VGG16 achieved accuracies of 0.8445, 0.8659, 0.8933, and 0.9497, respectively, and F1-scores of 0.8431, 0.8635, 0.8919, and 0.9494 in that order. Compared to these pre-trained models, the customized model MobileNet-BT achieved much higher scores in all metrics, with 0.9924 for the accuracy and 0.9924 for the F1-score. More information on the results can be found in Table I above. 

The revision of unfreezing all layers and modifying the classifier significantly improved the performance of medical image classification. While pre-trained models perform well in medical image classification, significant differences in dataset size and image types between generic and medical images require adjustments. Customizing the architecture and unfreezing layers of pre-trained models can dramatically enhance performance in medical image classification. MobileNet-BT attained an accuracy of 99.2%, in contrast to the plain MobileNetV2 model with an accuracy of 84.5%. This indicates the importance of further customizing and fine-tuning existing models to achieve significantly higher performance in medical image classification. 

Fig.2, 3, 4, 5, and 6 illustrate the loss and accuracy over epochs. MobileNet-BT achieved its best model at the 10th epoch, while others achieved their best models around or after the 20th epoch. This also indicates that the new model, MobileNet-BT, not only improves accuracy and other metrics such as F1-score but also makes predictions more efficiently.




§ CONCLUSION


Four pre-trained models were tested on the brain tumor dataset, resulting in accuracies of 0.8445, 0.8659, 0.8933, and 0.9497, respectively. The new model, MobileNet-BT, which is based on MobileNetV2, achieved an accuracy of 0.9924. Meanwhile, fewer epochs are required to train the new model compared to the pre-trained models. This model mainly focuses on medical image classification tasks and achieves much higher accuracy in the brain tumor classification task than other pre-trained models. Even though the pre-trained models were trained with much larger datasets for specific tasks, especially medical images, a revised model and customized architecture are required to achieve better outcomes.


IEEEtran







 Paper Title: S-SYNTH: Knowledge-Based, Synthetic Generation of Skin Images.







A. Kim et al.

Office of Science and Engineering Laboratories
 
Center for Devices and Radiological Health
 
U.S. Food and Drug Administration
 
Silver Spring, MD 20993 USA 

elena.sizikova@fda.hhs.gov 

*-These authors contributed equally to this work.


S-SYNTH: Knowledge-Based, Synthetic Generation of Skin Images
    Andrea Kim* Niloufar Saharkhiz* Elena Sizikova* 

Miguel Lago Berkman Sahiner Jana Delfino Aldo Badano
    August 5, 2024
==========================================================================================================

§ ABSTRACT 
Development of artificial intelligence (AI) techniques in medical imaging requires access to large-scale and diverse datasets for training and evaluation. In dermatology, obtaining such datasets remains challenging due to significant variations in patient populations, illumination conditions, and acquisition system characteristics. In this work, we propose S-SYNTH, the first knowledge-based, adaptable open-source skin simulation framework to rapidly generate synthetic skin, 3D models and digitally rendered images, using an anatomically inspired multi-layer, multi-component skin and growing lesion model. The skin model allows for controlled variation in skin appearance, such as skin color, presence of hair, lesion shape, and blood fraction among other parameters. We use this framework to study the effect of possible variations on the development and evaluation of AI models for skin lesion segmentation, and show that results obtained using synthetic data follow similar comparative trends as real dermatologic images, while mitigating biases and limitations from existing datasets including small dataset size, lack of diversity, and underrepresentation.






§ INTRODUCTION

Robust and generalizable artificial intelligence (AI) for medical applications requires large datasets representative of intended populations and their subgroups, relying on a time-consuming and laborious annotation process by clinical experts (e.g. labeling pixel-level segmentation masks or identifying findings). In this work, we explore the potential of a skin simulation model and the practical utility of resulting synthetic images in development and evaluation of AI-based  techniques. 

Segmenting and labeling dermatologic images is time-consuming and challenging, and thus public dermatologic datasets generally contain few examples and may not equitably represent intended patient populations <cit.>. For instance, the majority of the publicly available dermatologic images are associated with lighter skin populations, and dark skin tones are typically under-represented <cit.>. This class imbalance has shown to negatively affect darker skin tones in skin lesion segmentation tasks <cit.>. Such challenges may introduce biases in training and evaluation of medical AI devices for dermatologic applications. 

To address these limitations, we propose to simulate dermatologic images with a knowledge-based approach <cit.>, where properties of skin and lesions, determined via detailed physics models, along with rendering conditions are systematically varied. We created S-SYNTH, an open-source skin simulation framework for generating synthetic dermatologic images, and show that subgroup analysis using S-SYNTH images closely mimics comparative performance trends in real-patient skin datasets, while mitigating effects of mislabeled examples and limitations of a limited patient evaluation dataset. Our contributions can be summarized as follows:


  * We describe S-SYNTH, an open-source, flexible framework for creation of highly-detailed 3D skin models and digitally rendered synthetic images of diverse human skin tones, with full control of underlying parameters and the image formation process[Code and supporting data are available at: <https://github.com/DIDSR/ssynth-release>]. 

  * We systematically evaluate S-SYNTH synthetic images for training and testing applications. Specifically, we show S-SYNTH synthetic images improve segmentation performance when only a limited set of real images is available for training. We also show comparative trends between S-SYNTH synthetic images and real-patient examples (according to skin color and lesion size) are similar.




§ RELATED WORK
    
A particular challenge in practical development and evaluation of dermatologic AI is the lack of labeled datasets. Of the few public labeled datasets that exist, only a fraction have fine grained annotations, such as pixel-level segmentation levels for lesions <cit.> or hair <cit.>. The annotation process is error-prone and can have large variations across truthers (particularly when datasets are collected for different purposes and no consistent labeling protocol is established). In part due to these annotation difficulties, there is a lack of transparency and potential bias within the datasets used for dermatologic AI studies. A recent investigation found that from a total of 70 unique studies that involved AI models for dermoscopic imaging, only about 7 (10%) included skin tone information for at least one dataset used <cit.>. Synthetically generated data has been explored to address these challenges. Generative models have been used to create synthetic skin lesions to tackle the class imbalance problem and improve the performance of the skin lesion classifiers <cit.> or enhance lesion segmentation <cit.>. Rezk et al. have shown that synthetic skin images generated using style transfer and deep blending improve skin lesion classification accuracy by increasing skin color diversity <cit.>. More recently, diffusion models have been used to generate synthetic images to improve skin disease classifier performance <cit.>. Nevertheless, use of AI to generate synthetic images is limited by the requirement of constant supervision and adjustment of hyperparameters <cit.>, presence of artifacts, such as skin hair or lighting effects, within the original training dataset <cit.>, and a lack of ability to represent all physical and morphological features of disease <cit.> and image acquisition systems (see <cit.> for more information). 





§ S-SYNTH: SKIN SIMULATION FRAMEWORK

Our approach for generating synthetic skin involves the construction of a 3D digital object model comprising of skin tissue (epidermis, dermis, hypodermis), blood network, hair, and a lesion, as shown in Fig. <ref>a. This process is implemented and automated in Houdini <cit.>, a software for 3D modeling, animation, and visual effects, via a Python API. Once created, each model is subsequently processed through a research-oriented rendering system, Mitsuba 3 <cit.>, to generate each synthetic rendering (Fig <ref>b). The approach can be automated to create large databases with controlled variation for various AI analysis applications ((Fig <ref>c)). In the following sections, we describe the digital skin model and rendering process.



 §.§ Digital Skin Model



  §.§.§ Skin Tissue
 Each skin sample is created from a multi-layer model, with each layer representing a component of skin tissue. Thicknesses of the topmost two layers are based on values reported in <cit.>: epidermis (20-150 μ m) and dermis (1-4 mm). To introduce geometric variability, we incorporated surface roughness (noise) into the top surfaces of epidermis, hypodermis, and papillary dermis (located between the epidermis and dermis). Roughness values were randomized within predefined constraints, enhancing the naturalistic appearance and diversity of the generated skin models.



  §.§.§ Blood Network and Hair

The blood network model is created by solving a shortest path problem on randomly distributed points within a tetrahedral mesh generated from a primitive cube. Points from the bottom of the cube, corresponding to the lower blood network capillaries, were designated as start points, while those from top of the cube, representing the upper capillaries, were designated as end points. To generate each blood network model, the numbers and positions of starting and ending points were randomized. Hair models were constructed by manipulating hair properties (density, length, distribution, thickness, and curvature) in Houdini to generate a diverse range of hair apprearances.




  §.§.§ Growing Lesion Model

We developed a probabilistic, growing lesion model that generates a 3D volume with stochastic growth of lesion shape and size, based on <cit.>. Growth starts with one active cell (i.e., voxel). We define active cells as those cells that exist and can grow at a given time point. Within each time point, we iterated over the active cells c and selected them for growth based on probabilities (see Fig.  <ref>). To control the growing direction, the probability of an active cell to grow outside the skin is set to extremely low, while the probability for inwards growing is set to be high. Probabilities were based on a Gaussian distribution G, and are updated in each time point for added variability.



To control for lesion shape irregularity, we set an irregularity probability for each cell C_p, which starts C_i number of recursive iterations of the same growing algorithm on that cell's location, independent of the regular growth. Irregular cells can themselves trigger another recursive growth up to a maximum of C_r recursions. We generated both regular  (C_p=0.0001) and irregular lesions (C_p=0.001).  Sample images of lesions and growth evolution can be seen in Fig  <ref>.  Additional details can be found in the Supplementary Material. Once created, the 3D lesion volume is inserted into the skin tissue model (see <cit.> for more details).



 §.§ Rendering and Image Formation




  §.§.§ Optical Material Properties

Melanosomes are the primary contributors to optical absorption in the epidermis. In the dermis, optical absorption is mainly attributed to blood, while, in the hypodermis, lipid and fat components are the predominant sources of optical absorption. Thus, each skin layer is assigned an  optical material, containing the index of refraction (IOR) and the spectral distribution of the absorption and scattering coefficients.  Following the methodology established by Jacques et al. <cit.>, we calculated the optical absorption properties for each the four distinct tissue types: epidermis, dermis, hypodermis, and blood. We model reduced scattering of tissue using μ'_s = a(λ/500)^-b, where λ is the light wavelength <cit.>. Finally, for each skin layer material, we define a surface scattering model using a bidirectional scattering distribution function (BSDF), based on <cit.>.



  §.§.§ Lighting and Camera
 To account for lighting variations, we render each skin model using a collection of High Dynamic Range Imaging (HDRI) images capturing various  environmental lighting conditions from <cit.>. The Volumetric Path Tracer is a rendering algorithm that simulates the paths taken by light as it interacts with the 3D scene. It is particularly effective for scenes with volumetric effects or materials with complex light interactions. Spectral multiple importance sampling is a technique used to address the spectral nature of light. In scenes with materials that exhibit variations in spectral properties (e.g., spectral absorption), this method accounts for the complex interactions of light with materials across different wavelengths, such as those encountered in biological tissues. A perspective sensor with Mitsuba's HDR Film is placed 1.5cm above the skin with a field of view (FOV) of 75 degrees, facing perpendicularly to the top surface of the epidermis. The Integrator (Mitsuba 3's plugin for the rendering technique) is set to Volumetric Path Tracer with spectral multiple importance sampling. We chose this specific configuration due to its ability to simulate spectrally-varying optical properties across varying skin tones in simulated lighting conditions. We use 124 samples per pixel (SPP) and render images at resolution 1024x1024 pixels. Each image and mask take about three minutes to generate on a GPU. Sample generated images can be seen in Fig. <ref>. 
 




 §.§ AI Device Description

To demonstrate the usage of S-SYNTH  for augmenting real patient data, we experiment with real and synthetic examples on the task of skin lesion segmentation. which is important for timely treatment decisions <cit.>. We rely on DermoSegDiff <cit.>, a state of the art diffusion-based skin segmentation model, to differentiate lesions from the background region. We used the “dsd_i01” configuration, with , , and default training parameters. Images were segmented at resolution 128x128, and results were evaluated with Dice coefficient. We trained one model per experiment and reported standard deviation over test images.



  §.§.§ Datasets
 For real-patient datasets, we used two publicly available skin lesion segmentation datasets: ISIC18 <cit.> (ISIC) and HAM10K <cit.> (HAM). The same pre-processing and data splits described in <cit.> were applied, where the ISIC and HAM images were divided into (7200, 1800, 1015) and (1815, 259, 520) for the baseline model (training, validation, test), respectively. We used the individual typology angle (ITA) metric <cit.> to estimate skin tone from the non-lesion area. Finally, to better match lesion sizes across real and synthetic examples, S-SYNTH images were cropped to ensure a variety of lesion sizes. With S-SYNTH, we generated 10,000 images with randomized model properties and lighting, and subsequently 19,965 testing images with controlled variation: 5,445 with blood variation, 9,075 with melanosome fraction variation, 3,630 with different lesion regularity, and 1,815 without hair.   


§ RESULTS AND DISCUSSION
    
We present segmentation performance based on three aspects: impact of training set composition comprising different ratios of real and synthetic images, impact of various physiological or rendering parameters of the synthetic images when used as test set, and impact of specific characteristics of synthetic images that are measurable on real images.



  §.§.§ Synthetic Data for Training
 
We systematically evaluated the effect of the training data composition on the real test set performance (similar to the popular Train-Synthetic-Test-Real (TSTR) protocol <cit.>) on both ISIC and HAM. As shown in Fig. <ref>a, performance improved when more real examples are available for training. When real images are replaced with synthetic images, the resulting performance is comparable to that of the baseline model trained on full ISIC (or HAM) training set (Fig. <ref>b). However, performance dropped when the training set was fully composed of synthetic images, presumably due to the domain shift between the real and synthetic images <cit.>. More importantly, each model that was trained on a particular subset of the ISIC (or HAM) and supplemented with synthetic images resulted in a higher Dice score than the model that was trained on the same subset of only the real images. Finally, when full training sets were supplemented with synthetic images (Fig. <ref>c), performance improved for ISIC, but saturated for HAM, likely because ISIC was too small to learn all modes of variation, which synthetic data helped address. We also found that performance for darker skin tones improved with the addition of synthetic examples (see Supplementary Material). 






  §.§.§ Synthetic Data for Testing
 
To check whether synthetic images can produce meaningful performance trends, we evaluated models trained on real examples using synthetic test images. Performance dropped for increasing blood fraction (Fig. <ref>a), and increasing melanosome fraction (Fig. <ref>b) of the S-SYNTH images. Higher melanosome fraction corresponds to darker skin, and the corresponding performance drop is a known bias in dermatologic AI <cit.>. Models tested on lesions with regular shape exhibited higher performance compared to the ones tested on irregular lesions (Fig. <ref>c). Similar to <cit.>, we found that presence of hair slightly decreased performance (Fig. <ref>d). 

To assess the similarity between the real and synthetic images in measurable characteristics, we evaluated performance changes as a function of skin color (ITA), lesion circularity, and lesion relative area. Although performance on synthetic images was lower compared to real images, similar trends were observed. Specifically, performance in both synthetic and real examples increased with lighter skin (Fig. <ref>a). As illustrated in (Fig. <ref>b), the impact of lesion circularity on model performance showed similar trends within the circularity range shared between the real and synthetic images, where more circular images were easier to segment. Finally, performance dropped with larger relative lesion areas (Fig. <ref>c), possibly due to less precise delineations in larger lesions in the training patient data. 







  §.§.§ Limitations

S-SYNTH is the first skin simulation framework that generates synthetic skin images with controllable variations. There are a number of limitations to our work. First, S-SYNTH does not model any specific disease presentation, and additional  work needs to be performed to evaluate the realism of the generated examples. Second, the current rendering techniques utilizes an RGB camera setup, and additional modifications allowing for multispectral imaging, when validated, might be useful for more advanced algorithms that use a broader variety of wavelengths. 



§ CONCLUSION AND FUTURE WORK
    
We presented S-SYNTH, a novel, knowledge-based simulation pipeline for synthetic generation of dermoscopic images and skin lesions. S-SYNTH procedurally generates multi-layer 3D skin models, with consideration to optical skin characteristics, and digitally renders synthetic images under realistic lighting conditions. Using S-SYNTH, we generated realistic and varied skin surface models for sub-surface scattering simulation in the context of skin lesion segmentation, and compared the resulting synthetic images to public dermatologic benchmarks. We showed that examples created using S-SYNTH can be used to augment limited real datasets and identify performance trends. 


unsrturl










§ SUPPLEMENTARY MATERIALS


 







 



 Paper Title: GalleryGPT: Analyzing Paintings with Large Multimodal Models.


















  Tongji University
  
  Shanghai
  China

  National University of Singapore
  
  
  Singapore
yi.bin@hotmail.com




  University of Electronic Science and Technology of China
  
  Chengdu
  China
shiwenhao16@gmail.com


The corresponding author.

  The Hong Kong Polytechnic University
  
  Hong Kong SAR
  China
dingyujuan385@gmail.com




 Singapore University of Technology and Design
  
  
  Singapore
zhiqiang_hu@mymail.sutd.edu.sg




  Tongji University
  
  Shanghai
  China
zh_wang@hotmail.com




  University of Electronic Science and Technology of China
  
  Chengdu
  China
yang.yang@uestc.edu.cn




  National University of Singapore
  
  
  Singapore
seekiong@nus.edu.sg




  Tongji University
  
  Shanghai
  China

  University of Electronic Science and Technology of China
  
  Chengdu
  China
shenhengtao@hotmail.com











§ ABSTRACT 
  Artwork analysis is important and fundamental skill for art appreciation, which could enrich personal aesthetic sensibility and facilitate the critical thinking ability. Understanding artworks is challenging due to its subjective nature, diverse interpretations, and complex visual elements, requiring expertise in art history, cultural background, and aesthetic theory. However, limited by the data collection and model ability, previous works for automatically analyzing artworks mainly focus on  classification, retrieval, and other simple tasks, which is far from the goal of AI. To facilitate the research progress, in this paper, we step further to compose comprehensive analysis inspired by the remarkable perception and generation ability of large multimodal models. Specifically, we first propose a task of composing paragraph analysis for artworks, , painting in this paper, only focusing on visual characteristics to formulate more comprehensive understanding of artworks. To support the research on formal analysis, we collect a large dataset PaintingForm, with about 19k painting images and 50k analysis paragraphs. We further introduce a superior large multimodal model for painting analysis composing, dubbed GalleryGPT, which is slightly modified and fine-tuned based on LLaVA architecture leveraging our collected data. We conduct formal analysis generation and zero-shot experiments across several datasets to assess the capacity of our model. The results show remarkable performance improvements comparing with powerful baseline LMMs, demonstrating its superb ability of art analysis and generalization. The codes and model are available at: <https://github.com/steven640pixel/GalleryGPT>.




<ccs2012>
   <concept>
       <concept_id>10010405.10010469</concept_id>
       <concept_desc>Applied computing Arts and humanities</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10010147.10010178.10010179.10010182</concept_id>
       <concept_desc>Computing methodologies Natural language generation</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>


[500]Applied computing Arts and humanities
[500]Computing methodologies Natural language generation
























GalleryGPT: Analyzing Paintings with Large Multimodal Models
    Heng Tao Shen
    Received 9th July 2024; accepted 13th July 2024
============================================================









§ INTRODUCTION

Artwork analysis and composition are integral aspects of art appreciation and creation, often requiring a deep understanding of artistic techniques, styles, and historical contexts. In the past decades, AI systems have been evolving rapidly and demonstrating remarkable success in many fields, even surpass humans <cit.>. However, it still cannot understand and analyze an artwork like humans since it involves very high-level joint understanding of culture, symbolism, abstractionism, and other aesthetics knowledge, beyond the basic semantic understanding of objects, attributes and relations in natural image understanding. Motivated by the superior ability of deep learning, researchers have employed several advanced techniques, such as convolutional neural networks (CNN) <cit.>, recurrent neural networks <cit.>, and Transformer <cit.> in  style classification, object detection, multimodal retrieval, art visual question answering, and artwork captioning <cit.>. Despite these advancements, there is still a lack of research on composing comprehensive and in-depth analysis for artworks, limited by the model capacity of visual perception and language generation.



The emergence of large foundation models, such large language models (LLMs) <cit.> and large multimodal models (LMMs) <cit.>, has facilitated the progress across numerous research areas, including text summarization, open-ended question answering, and long-context reasoning, and led to significant advancements <cit.>. This also makes it possible to enable machine to perceive and understand visual content and generate detail descriptions, such as visual storytelling <cit.>. However, despite the achievements on natural image understanding, we observe that existing LMMs still cannot comprehend the high-level concepts and generate comprehensive analysis for artworks. As the example illustrated in Figure <ref>, we test several powerful LMMs, , GPT-4V <cit.>, Gemini <cit.>, and ShareGPT4V <cit.>, with a nameless painting[We choose a nameless painting to avoid the knowledge about the painting has been seen during pre-training and Supervised-Fine-Tuning (SFT), since the knowledge about famous ones is memorized by the LLMs. This nameless painting is downloaded from: <https://darksky.org/app/uploads/2015/11/8_Florinus_Astronomy.png>] by asking them to compose a paragraph of formal analysis. From the outputs, we can observe that the GPT-4V and Gemini wrongly recognized this painting to another, and then give the analysis based on the knowledge recalled from their language-part memories, which means they function as LLMs in this part. We call this phenomenon as “LLM-biased visual hallucination”. This phenomenon suggests that in this task, existing LMMs tend to first recognize the given painting is and then give analysis accordingly, while do not focus on the visual content of the painting at the stage of analysis generation. Such a recognize-then-analyze procedure highly relies on the accuracy of the recognition, and will fail when the given painting is unknown. 

Although we have noticed that 

GPT-4V tries to analyze this painting based on the visual content (green part shown analysis shown in Figure <ref>), it still cannot completely escape from the recognize-then-analyze procedure. 

Meanwhile, some effort has been devoted to enhance the visual understanding of LMMs. For example, ShareGPT4V <cit.> contributes a dataset with high-quality image and description pairs to fine-tune the LLaVA model <cit.>. 

Despite of the effectiveness, these methods still fail to make professional artistic, aesthetic and technical analysis on artworks or paintings,  

meaning that existing LMMs so far cannot satisfy the requirements of artwork analysis. They may face particular challenges to get generalized into unknown or unseen subjects, thereby calling for better solutions.  










To fill the research gap, this paper focuses on generating formal analysis for painting images, relying on the LMMs but pushing them to perceive and comprehend the artistic skills or other professional visual aspects shown in an artwork itself. A comprehensive artwork analysis includes many parts, , introduction, cultural, formal analysis, historical context, interpretation, . Although most of artwork analysis may be done relying only on the external knowledge or subjective opinions, the “form” of the artwork, including color, composition, line, shape, light and shadow, and other visual aspects , still requires vision-based understanding. Therefore, it is necessary to develop a professional LMM specialized for making formal analysis for artworks. 


However, there does not exist any formal analysis dataset. It is also time- and labour-expensive to make annotations because writing a formal analysis for an painting requires professional expertise in art analyzing. Motivated by the aforementioned observed LLMs bias by GPT4 and Gemini, we try to access the knowledge memorized in LLMs to produce analyses for known artworks. Specifically, we first collect about 19k famous painting images and corresponding meta data from the Internet. Then, apply LLMs to provide a paragraph analysis only focusing on visual characteristics based on the title and artist of the painting, thereby generating the formal analysis. We also prompt the LLMs to compose the formal analysis from some specific form perspectives, such as composition, color, light and shadow, to enhance the richness and diversity of the data. We finally obtain about 50k formal analyses for the paintings we collect and name this dataset as PaintingForm.


Leveraging the PaintingForm dataset, we present

GalleryGPT, a large multi-modal model with a LLaVA architecture fine-tuned based on ShareGPT4V-7B <cit.>. As ShareGPT4V is boosted for image understanding and performs well in visual description generation, we freeze the parameters in vision encoder to retain its superb visual perception ability. Meanwhile, we add a LoRA component to LLM to learn the  specific analyzing patterns of paintings. To evaluate the effectiveness of our GalleryGPT, we conduct zero-shot learning on several classic painting analysis datasets, including AQUA <cit.>, ArtQuest <cit.>, and ArtQuest-Type <cit.> for art visual question answering, and ArtBench <cit.> for style classification. The results show that our GalleryGPT outperforms several off-the-shelf and adaptive LMMs, demonstrating the superiority of our collected data and impressive performance of GalleryGPT.

In summary, the contributions of this work are as follows:

    
  * We propose to empower the perception ability of LMMs for subtle and specific visual characteristics of artworks, and introduce the task of generating formal analysis to enable the ability by supervised fine-tuning.
    
  * To support the research, we contribute a large-scale and high-quality dataset PaintingForm, acquired by two powerful LLMs, , GPT-4 and Gemini, based on the learnt knowledge about famous paintings. To avoid the leakage of prior knowledge, we ask the LLMs only focus on visual characteristics and do not mention the title and artist in the formal analysis annotation.
    
  * Leveraging the collected dataset, we devise an advanced large multimodal model for painting formal analysis generation, dubbed GalleryGPT, which employs ShareGPT4V as backbone. We evaluate its ability on several painting analysis tasks, and the results demonstrate the impressive performance of it.








§ RELATED WORKS






 §.§ Large Language Models (LLMs)



In the past decade, language modelling has been evolving rapidly and achieved impressive progress <cit.>. Transformer <cit.> employed multiple attention blocks and positional embedding to accelerate the recurrent models and made breakthrough in language models. Delvin <cit.> designed a bidirectional transfomer to learn the contextual embedding of words and made the beginning the era of pre-training language model (PLM) <cit.>. With the large scale tokens pre-training, deep neural models, especially the large language models (LLMs) <cit.> based on the Transformer <cit.> structure, demonstrate superb understanding and generation ability, as well as generalizing to downstream tasks, even without any fine-tuning. LaMDA<cit.> focused on conversational applications and aims to generate more natural and logically-rich dialogue text. InstructGPT<cit.> designed an effective fine-tuning method that allows LLMs to operate according to desired instructions, leveraging Reinforcement Learning from Human Feedback(RLHF). It incorporates humans into the training loop using carefully designed annotation strategies. With the instruction fine-tuning based on large foundation models, LLMs, especially the ChatGPT system <cit.> of OpenAI, also demonstrate emergent capacity of generation <cit.>.  The LLaMA<cit.> model released by Meta, with its open-source nature and smaller parameter size, has provided an opportunity for many researchers to participate in large language model research and led to rapid research progress in LLMs.



 §.§ Large Multimodal Models (LMMs)

The blooming of large language models has attracted a great amount of research attention on vision-language interaction and injecting visual knowledge into LLMs. As a paradigm of visual language modal alignment, CLIP <cit.> implemented contrastive learning on extensive image-text pairs. Subsequent improvements <cit.> over CLIP utilized enhanced data strategies with greater data diversity for basic visual tasks. Recent research has increasingly focused on pre-traning alignment and visual instruction tuning on top of the LLMs for more complex tasks such as visual question answering and reasoning. MiniGPT-4 <cit.> demonstrated capabilities in image-text dialogues by aligning queried visual feature with text and feeding queried embedding to LLM. Other prominent examples including LLaVA <cit.>, Qwen-VL <cit.>, InstructBLIP <cit.>, and ShareGPT4V <cit.> interacted visual features with LLM using a learnable projector or query embeddings, which focus on utilizing more and high quality pre-training and fine-tuning data to understand complex instructions. mPLUG-Owl <cit.>, Shikra <cit.>, and KOSMOS-2 <cit.> introduced grounding data types and new modularization training to minimize hallucinations and enhance grounding ability. Despite these advancements, exploration for quality and format of images-instructions highlights a critical area for future large multimodal models improvement.









 §.§ AI for Art Analysis

With the great success of deep learning in CV and NLP in the past decade, AI for art analysis has also been evolving with rapid progress. Early works most depended on hand-crafted features and explored the classification and recognition problem <cit.>. With the great success of pre-trained language models (PLM) <cit.> and visual-language pre-training (VLP) <cit.>, research on artwork analysis also achieves significant progress. CLIP-Art <cit.> leveraged the image-text pairs of artworks in SemArt <cit.> to fine-tune the CLIP and gains impressive improvements. Garcia  <cit.> contributed a dataset of art visual question answering, named AQUA, and proposed a knowledge-based VIKING model, which achieves the best performance. Bleidt  <cit.> pointed out there exists language bias hidden in the question-answer pair, which may induce the model to ignore the visual information. To address this issue, they further proposed a strategy to eliminate the bias and introduced an ArtQuest dataset and implemented PrefixLM to learn the patterns between paintings and questions.


Despite these simple tasks investigated in deep learning era, people also expect the AI system could provide comprehensive analysis for artworks, which may benefit the art education and assist human writing the commentary. Recently, LLMs and LMMs have been demonstrating superb understanding and generation ability in many areas, which raise the potentials to compose comprehensive analysis for artworks. However, we argue current LMMs might be biased by the knowledge memorized in language and cannot generalized to nameless artworks. In this work, we try to implement formal analysis with LLMs to make them focusing on visual elements in art analysis.



§ DATA COLLECTION












 §.§ Primary Philosophy


A successful artwork analysis, as been discussed before, needs to cover multiple types of content including background introduction, formal analysis, historical context, , most of which rely on external knowledge and subjective opinions except formal analysis.
As we have known, existing of-the-shelf LMMs have been capable of providing factual information and description with relevant knowledge while tend to ignore visual analysis, we want to enhance the existing artwork analysis models to focus more on visual elements. The research purpose in this work is to make the developed model to emphasize on visual comprehension rather than recognizing the artwork and retrieving knowledge from their language memory, in other word, to twist the recognize-then-analyze procedure of existing LMMs on artwork analysis which may cause “LLM-biased visual hallucination”. 

To support the development of such a professional LMM specialized for artwork analysis, we try to construct a large-scale artwork analysis dataset, including images and corresponding analysis, so that we can fine-tune the pre-trained LMMs. More importantly, to push the LMMs to focus on visual comprehending of the artworks through fine-tuning, we need high-quality formal analysis annotations in our dataset. However, manually annotating artworks with comprehensive analysis requires professional expertise in artwork analyzing and it is hard and expensive to recruit so many professional art critics. Besides, even with experts, it is still time-consuming to annotate large number of artworks. 



Based on these motivations and challenges, we collect a painting dataset, named PaintingForm, consisting of paintings and associated formal analyses annotations for each painting. Since manual labeling is affordably expensive in terms of both timing  and economic cost, 

we elaborately leverage the powerful LLMs to produce formal analysis on famous paintings, ensuring they have enough knowledge to support them producing high-quality analysis. 

The overall pipeline of our data collection process is shown in Figure <ref> and more details are described in subsequent sections.








 §.§ Painting Source


With the development of digital technology in recent years, large collections of artworks have been digitized and stored, and easily to accessed via the Internet. We focus on famous paintings and choose 1st Art Gallery[<https://www.1st-art-gallery.com/>] as our painting source. To make the LMMs, , GPT-4 and Gemini, able to provide accurate and comprehensive formal analyses, we choose the paintings of 500 Most Popular Paintings[<https://www.1st-art-gallery.com/most-popular-paintings.html>] and Most Popular Artists[<https://www.1st-art-gallery.com/browse-by-artist-a-z.html>. Note that here we only use the most popular artists, not all, resulting in 48 artists in total.] to ensure the LLMs knowing the paintings. With such condition, we obtain 19,295 paintings (including 18,795 of most popular artists). To ensure the LLMs know the paintings, we first ask Gemini to answer if it knows the painting with title and artist name. We also filter out some paintings without certain title and annotated as “unknown”, and obtain 18,526 paintings in the end. We select 5000 less popular paintings for test and reserve 13,526 for training,  test samples of which are identified by the wishlist count from source website, and the distribution across artists is also considered. We illustrate the statistics of artist distribution of the Most Popular Artists[We do not include the 500 most popular paintings in this statistics because about 150 artists only contribute one painting in the whole gallery.] in Figure <ref>. From the statistics, we observe that the dataset includes most paintings of Vincent Van Gogh, resulting in 1458 retained and 225 filtered, and Jacques Louis David with the fewest paintings included in the dataset, about 100 paintings.
We have also verified that all the paintings are free to use without copyright concerns, as provided by the website[<https://www.1st-art-gallery.com/copyrights.html>, The copyright contents are as [Accessed on 4 April, 2024.]: “”].




 §.§ Formal Analysis Annotation

As aforementioned, we employ powerful LLMs to generate high quality formal analysis, after ensuring the paintings are known by the LLMs.  
Note that we only provide the LLMs the title and artist name of the paintings, and do not input any visual information, , the painting image, to the LLMs. 
Specifically, we employ two powerful LLMs, GPT-4 and Gemini in this work, to retrieve the learnt knowledge with the title and artist name of a certain painting and generate a paragraph of formal analysis only focusing on visual characteristics. To make the formal analyses more diverse, we ask the LLMs to generate two-level formal analyses: 1) overall formal analysis, and 2) formal analysis from a certain perspective, , color, composition, and . To avoid the “LLM-biased visual hallucination”, we ask the LLMs do not mention the title and artist name in the generated formal analysis. In other words, one cannot easily identify the corresponding painting solely depending on a specific formal analysis. For the perspective specified formal analysis annotation, we first ask GPT-4 and Gemini to provide up to 5 most important perspectives of the given painting independently, and utilize the intersection of GPT-4 and Gemini predictions as the final perspectives. The statistics of perspectives is shown in Figure <ref>, from which we observe that Composition, Light and Shadow, Color Palette, Form and Shape are the most common perspectives of the paintings. Finally, based on the selected perspectives, we employ LLMs to annotate the formal analysis only focusing on a specific perspectives similar with the overall setting.





§ THE PROPOSED GALLERYGPT

To verify if our proposed PaintingForm dataset could empower LMMs more superior painting analyzing ability, we develop a large multimodal model, GalleryGPT, based on ShareGPT4V-7B that has demonstrated  impressive performance across several multimodal tasks. We conduct supervised fine-tuning (SFT) on our PaintingForm dataset to enable the LLMs to analyze paintings focusing on visual elements.



 §.§ Architecture

As pointed out before, our goal is enabling LMMs to analyze artworks focusing more on visual elements, rather to design a new architecture or model. Therefore, we introduce the GalleryGPT employing ShareGPT4V-7B as backbone, which follows the LLaVA <cit.> architecture and consists of three components: 1) The visual encoder, which is a vision transformer (ViT) <cit.> borrowed from the CLIP-Large <cit.>. Similar with CLIP-Large, the visual encoder takes 336*336 shape as input size and divides it into 14 patches, resulting in 576 input tokens; 2) The projector, a two-layer MLP to project the visual representation into the language semantic space, LLMs space in specific; 3) The LLM, which is based on Vicuna-v1.5 <cit.> and LLaMA2 <cit.>, employing the decoder-only architecture. For the whole setting, we follow ShareGPT4V-7B focusing on the 7B model. Besides, to keep the superior visual perception and describing ability of ShareGPT4V-7B, we slightly modify the LLM in our GalleryGPT. Specifically, we add several LoRA <cit.> modules to learn the formal analysis specific patterns and freeze the LLM in ShareGPT4V-7B to keep its superb  content  describing ability.



 §.§ Supervised Fine-Tuning

Numbers of previous works <cit.> have demonstrated that based on billions or trillions of tokens pre-training and elaborate supervised fine-tuning, the LLMs exhibit creative emergent ability and is able to generalize to multiple tasks without further training. Following this inspiration, we implement supervised fine-tuning (SFT) with our LLM-generated formal analyses, paired with the corresponding painting. 
During SFT stage, we set the learning rate as 2e-5, batch size as 16, and fine-tune 10k steps on an A100 GPU with 80G memory. For the LoRA module, we set the lower rank as 128 and the alpha is 256.
Since our analysis texts do not contain any identifying information of the paintings, the LMMs cannot retrieve the learnt knowledge to associate with the paintings. Instead, with our SFT optimization, the LMM tries to perceive the subtle elements and art skills presented in the painting, and matches them with the corresponding descriptions in the formal analysis. Through such fine-tuning, our GalleryGPT could be empowered artwork analyzing ability and generalized to other art analyzing tasks, , style classification.



§ EXPERIMENTS



 §.§ Formal Analysis Generation

Since our data and optimization goal are for painting formal analysis generation, we first directly evaluate the quality of generated formal analyses on our reserved test set, with 5000 paintings. We also test with several popular and powerful open-source LMMs, including LLaVA-1.5 <cit.>, Qwen-VL-Chat <cit.>, and ShareGPT4V[Here we implement 7B model for fair comparison for all baseline and omit 7B in the table. Note that in <cit.>, ShareGPT4V indicates the dataset and ShareGPT4V-7B is the model, but in this table we omit 7B for unifying all the model names.] <cit.>, and all the LLMs employ the same prompt: “Please compose a coherent paragraph of formal analysis focusing on visual characteristics”.  We combine the formal analyses annotated by GPT-4 and Gemini for each painting to formulate the ground truth descriptions. Finally, we employ image captioning metrics to evaluate the generated formal analyses because formal analysis is also a kind of description. As the evaluation results shown in Table <ref>, we observe that our GalleryGPT outperforms all the other LMMs with a remarkable improvement. Among all the baseline LLMs, LLaVA-1.5 performs the worst and is significantly lower than others, while Qwen-VL and ShareGPT4V achieve similar performance. From these observations, we can conclude that, benefiting from the SFT with PaintingForm, the GalleryGPT achieves impressive improvements comparing with ShareGPT4V-7B, which mainly focuses on natural image-text pairs (only a few art data). This verifies the effectiveness of our high quality art analysis data collections.

















 §.§ Generalizing to Other Art Analysis Tasks


LLMs and LMMs have demonstrated superior generalizing ability to many downstream tasks. To verify the generalization of our GalleryGPT, we conduct visual question answering and style classification for paintings on several existing datasets. The details of the dataset are as following:

    
  * AQUA <cit.>: is an Art Question Answering dataset based on SemArt <cit.>. The question-answer pairs are generated by powerful question generation models based on paintings and comments provided in SemArt. The dataset contains 29,568, 1,507, and 1,270 samples in training, validation, and test splits, respectively.
    
  * ArtQuest <cit.>: is the debiased version of AQUA, including 6414 test cases, which mainly focuses on eliminating the bias hidden in language. With such debias operation, most question cannot be answered without visual content. 
    
  * ArtQuest-Type <cit.>: is simple setting of ArtQuest, which requires the model only answer the type of the painting. The test set contains 1069 samples.
    
  * ArtBench <cit.>: is originally for Artwork generation. We random sample 1000 paintings from the test set (10000 paintings in total), and use the provided style labels for classification.


We exhibit the results in Table <ref>, from which we can see that our GalleryGPT also significantly outperforms all the baseline LMMs and demonstrates its generalizing ability for downstream art analysis tasks. An interesting observation is that LLaVA-1.5 performs much better than Qwen-VL-Chat, showing an opposite result to the task of formal analysis generation. We have checked and analyzed the results, and hypothesize that may come from the output format mismatching. Since all the three QA datasets follow the open-ended answering setting, and then extract the matching strings from the outputs. Another piece of evidence is that Qwen-VL-Chat achieves similar performance to LLaVA-1.5 when the output strictly adheres to the given labels.

To comprehensively analyze the ability of our GalleryGPT, we also compare it with several state of the arts and show the results in the top block in Table <ref>, which are specific trained on the corresponding datasets. SoTA-1 denotes the models without pre-training language model (PLM), , BERT, and SoTA-2 denotes the models leveraging PLM features or fine-tuning based on PLMs. From the results we can see, almost all the current LMMs only achieve the performance around SoTA-1 on AQUA, , specifically training on this dataset, while they can significantly outperform SoTA-1 on ArtQuest. This reason may come from the language bias hidden in AQUA dataset <cit.>. For all the dataset, the performances of LMMs are far from the SoTA-2, which means the generalization ability of LMMs on art analysis is still weak and there exists a large space for them to get improved. Our hypothesis on such results is that  artworks may contain high-level and abstract concepts to be perceived more subtly, which also motivates us to make further endeavour on this in the future work.






 §.§ Multimodal Benchmarks

Since GalleryGPT is implemented based on ShareGPT4V-7B, it inherently remains an LMM. Therefore, we also conduct experiments on several LLM benchmarks and compare it with baseline LLMs[Instead of listing all the  baselines in <cit.>, here we only list the superior and representative ones for comparison.] to evaluate the multimodal understanding ability. As the comparison results illustrated in Table <ref>, we can see our GalleryGPT exhibits comparable performance with ShareGPT4V, , achieving improvements and reducements in slight fluctuations. In other words, our GalleryGPT, further fine-tuned with elaborately collected painting-analysis pairs, not only achieves better painting analysis performance, including formal analysis generation (Table <ref>) and other downstream tasks (Table <ref>), and also retains its superior multimodal understanding ability. These observations further verify the effectiveness and contributions of our collected PaintingForm and developed GalleryGPT.




UTF8bsmi








 §.§ Qualitative Analyses

To visually and straightforwardly compare our GalleryGPT with other LMMs, inncluding LLaVA-1.5, ShareGPT4V-7B, and GPT-4V, we test some unpopular paintings in our test split. As illustrated in Figure <ref>, we observe that LLaVA tends to describe the factual content (in purple color) presented in the painting, and fails to analyze the paintings from the perspective of art criticism. ShareGPT4V-7B, the backbone of our model, also pays more attention on the content description, because the SFT data it utilized for fine-tuning is focusing on describing the images in detail. Thanks to the small subset of art data in ShareGPT4V, it also exhibits some capacity for art analysis, albeit limited to general analyses and overlooking the subtle visual characteristics. GPT4V, as the most famous and capable LMM at current stage, demonstrates much better analyzing ability on paintings in terms of their form, while still fails to capture some subtle artistic aspects, such as composition and depth. Obviously, our GalleryGPT demonstrates superb capability of comprehensively analyzing the artworks, which not only briefly describe the factual content of the painting, but also focuses more attention on analyzing the subtle artistic elements, including color, light and shadow, depth, composition, and perspective. These observations have definitely verified the superiority of our GalleryGPT for artwork analyzing.

We also investigate the dialogue capability of our GalleryGPT. We show several dialogue examples in Figure <ref>. The examples demonstrate that our GalleryGPT is able to follow the user intention in the conversation. As shown in the second case, for instance, we ask it to give “essential aspects” for formal analysis, it just provides us a brief analysis with essential content, which is much shorter than the one provided in Figure <ref>. Besides, we further explore the multilingual capabilities of GalleryGPT, even though we have not provided any multilingual painting analyses for supervised fine-tuning[Actually, the foundation models, LLaMA and LLaVA, are pre-trained and fine-tuned with several multilingual corpuses.]. As shown at the bottom in Figure <ref>, we chat with GalleryGPT in Chinese[A reviewer pointed out that the “前途” and “后途” are typos and should be “前景” and “后景”, since this analysis is generated by our GalleryGPT and we have not modified it. As the English translation in Figure. 6 in Supplementary, the meanings are the same with the suggestions of the reviewer. We sincerely appreciate the reviewer providing such careful and helpful comments and suggestions.], it still generates high quality answer focusing on “composition” and exhibits strong multilingual ability.


In summary, these qualitative comparisons and dialogue examples have verified the superior art analysis ability of our GalleryGPT, as well as illustrating the quality of our PaintingForm dataset. For more examples of this part, qualitative comparison and chat conversation, could be found in Supplementary.






§ CONCLUSIONS AND FUTURE WORKS

In this work, we targeted at artwork analyzing with large multimodal models (LLMs). We first tested several LLMs and pointed out that current LLMs may suffering from “LLM-biased visual hallucination” issue, resulting in weak generalization ability to nameless paintings. To make the LMMs to analyze artworks focusing on visual elements and easy to generalize, we proposed to conduct SFT of LMMs with formal analysis. To support this research, we first elaborately designed an LLM-based data collection pipeline to construct high quality painting-analysis pairs. We also employed ShareGPT4V to implement SFT on the collected data, derived our GalleryGPT. We conducted extensive experiments to verify the effectiveness regarding to formal analysis generation, generalizing to down stream tasks, and LMM benchmarking. The results demonstrated the effectiveness of the collected data and introduced GalleryGPT.

For future works, on one hand, we will step further to investigate the generalization issue mentioned in Section <ref> to explore the ability of LMMs in art analyzing. For example, devising a ChatGPT-like art assistant to help more people appreciate or learn to appreciate  artworks. On the other hand, we only collect paintings in this work, while artworks consist of multiple types, , ceramics (paintings on a curved surface) and sculptural in 3D. Therefore, we will try to make the research more widely  and empower LMMs to assist human in artwork analyzing, , drafting formal analysis, classifying the unseen artworks, . We also hope that our work can inspire more researchers to explore AI, especially for LMMs, within the field of art analysis.






This work is supported by the National Natural Science Foundation of China under grant 62102070, 62220106008, and 62306065. This research/project is supported by the National Research Foundation, Singapore under its Industry Alignment Fund – Pre-positioning (IAF-PP) Funding Initiative. Any opinions, findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.


We also sincerely thank all the ACs and reviewers for their efforts on our work and appreciate the useful comments for improving it.





ACM-Reference-Format














































§ INTRODUCTION

Due to the page limit, we omit some details in the main paper, , the detail prompts for data collection, and only illustrate one example for some parts.
In this supplementary materials, we will first give the prompting details and then illustrate more examples to make the qualitative analysis more comprehensive and convincing.



§ DETAIL PROMPTS FOR DATA COLLECTION

As described in Section 3.1 and the pipeline shown in Figure 2, our formal analysis annotation process consists of three stages: 1) verifying if the LLMs know the painting, 2) extracting essential elements presented in the painting, and 3) generating a paragraph formal analysis for each painting. For each stage, we use different instructions to prompt the LLMs. Limited by the space, we omit the details prompts in the main part, and illustrate them here for details. For the third stage, we ask the LLMs to annotate two kinds of formal analysis: overall formal analysis (prompt for stage 3-1) and specified aspect formal analysis (prompt for stage 3-2).



        
    
    
    0.85
    Prompt for Stage 1:

    You are a professional art critic. 

    Do you know the painting titled [Painting Title] by [Artist Name]? Respond with only 'YES' or 'NO'.
    


    
    0.85
    Prompt for Stage 2:

    You are a professional art critic. 

    Please select up to five essential aspects for the formal analysis of the [Painting Title] by [Artist Name] from the provided list. You may choose fewer if preferred. 
        
    The list is: ["Composition", "Color Palette", "Line Quality", "Texture", "Light and Shadow", "Perspective", "Form and Shape", "Movement and Gesture", "Symbolism and Iconography", "Scale and Proportion"]. 

    Please list the aspects in descending order of importance in a LIST.
        Your output should be in the format: "The essential aspects are: [YOUR SELECTED CHARACTERISTICS].
    



    
    0.85
    Prompt for Stage 3-1:

    Overall formal analysis annotation:

    You are a professional art critic. 

        Please compose a coherent paragraph, approximately [Number of Words] words, providing a professional formal analysis of the [Painting Title] by [Artist Name] ONLY focusing on the VISUAL CHARACTERISTICS of this painting.

        DO NOT mention the painting title and artist's name in your analysis. 

        You may use conjunctions as needed to ensure coherence in semantics and logic in your analysis.
    


    
    0.85
    Prompt for Stage 3-2:

    Specified aspect formal analysis annotation:

        You are a professional art critic. 

        Please write a paragraph about [Number of Words] words of formal analysis of the [Painting Title] by [Artist Name] ONLY focusing on the [Specified Aspect]. 

            DO NOT mention the painting title and artist's name in your analysis.
    
    








§ MORE CASES FOR QUALITATIVE ANALYSIS









 §.§ More Cases for Comparison

We first illustrate more cases for comparison with other LMMs in Figure <ref>, Figure <ref>, and a failure case in Figure <ref>. From the former two cases, we can see the similar observations in Figure 5 in main part. Here we focus on the failure case in Figure <ref>, from which we can observe that our GalleryGPT incorrectly recognized a vase as a candlestick (text with red color in Figure <ref>). This implies that the vision perceptron in our GalleryGPT needs to be improved to ensure the accurate visual encoding.

Then we illustrate more dialogue case with our GalleryGPT in Figure <ref> and  <ref>. From Figure <ref>, we can see that when the GalleryGPT is generating overall formal analysis, it would briefly analyze the main visual elements in the painting, while when we ask it to generate formal analysis focusing on specific aspect, , Light and Shadow in Figure <ref>, it could give more in-depth analysis from this aspect and also consistent with the brief one.




 §.§ The English Translation for The Chinese Case

Finally, we give the English translation for the case in Chinese illustrated in Figure 6. To further investigate the multilingual capacity, we also test the same chat content in English with our GalleryGPT, and it outputs similar formal analysis in high-level semantics and concepts, but very different words and aspects in low-level semantics. This implies that our GalleryGPT is able to understand and generate the formal analysis with multiple languages.






































 Paper Title: DECIDER: Leveraging Foundation Model Priors for Improved Model Failure Detection and Explanation.



















DECIDER: Leveraging Foundation Model Priors for Improved Model Failure Detection and Explanation
    Rakshith Subramanyam* rakshith.subramanyam@axio.ai 

       Axio.ai Kowshik Thopalli* thopalli1@llnl.gov 

       Lawrence Livermore National Laboratory Vivek Narayanaswamy* narayanaswam1@llnl.gov 

           Lawrence Livermore National Laboratory 
           Jayaraman J. Thiagarajan jjthiagarajan@gmail.com 

       Lawrence Livermore National Laboratory 
       
    August 5, 2024
==================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Reliably detecting when a deployed machine learning model is likely to fail on a given input is crucial for ensuring safe operation. In this work, we propose DECIDER (Debiasing Classifiers to Identify Errors Reliably), a novel approach that leverages priors from large language models (LLMs) and vision-language models (VLMs) to detect failures in image classification models.
DECIDER utilizes LLMs to specify task-relevant core attributes and constructs a “debiased” version of the classifier by aligning its visual features to these core attributes using a VLM, and detects potential failure by measuring disagreement between the original and debiased models. In addition to proactively identifying samples on which the model would fail, DECIDER also provides  human-interpretable explanations for failure through a novel attribute-ablation strategy. Through extensive experiments across diverse benchmarks spanning subpopulation shifts (spurious correlations, class imbalance) and covariate shifts (synthetic corruptions, domain shifts), DECIDER consistently achieves state-of-the-art failure detection performance, significantly outperforming baselines in terms of the overall Matthews correlation coefficient as well as failure and success recall. Our codes can be accessed at <https://github.com/kowshikthopalli/DECIDER/>

  




Failure Detection, Vision-Language Models, Large-language Models

* equal contribution




§ INTRODUCTION

A crucial step in ensuring the safety of deployed models is to proactively identify if a model is likely to fail for a given test input. This enables the implementation of appropriate correction mechanisms without impacting the model's operation, or even deferring to human expertise for decision-making. While failures in vision models can be attributed to a variety of factors, the most significant cause is the violation of data distribution assumptions made during training <cit.>, which is the focus of this work. In general, data comprises both task-relevant core attributes and irrelevant nuisance attributes, and they are never explicitly annotated. Consequently, models can fail to generalize if (i) the training data contains spurious correlations (to nuisance attributes) that do not appear at test time, (ii) class-conditional distribution of nuisance attributes can arbitrarily change between train and test data (e.g., patient race imbalance in clinical datasets), or (iii) novel attributes emerge only at test time (e.g., style changes). Note that, when the class-conditional distributions of core attributes themselves change between train and test data, it leads to the more challenging scenario of concept shifts, and is not considered in this work. Nevertheless, detecting failures across all these scenarios is known to be challenging <cit.>, and hence there has been a surge in research interest <cit.>.

We begin by acknowledging that it is not only difficult, but also inefficient, to describe such nuisance attribute discrepancies solely using visual features. In this regard, we explore the utility of large language models (LLMs) and vision-language models (VLMs) in characterizing data attributes through a combination of visual and natural language descriptors. Subsequently, one can leverage these descriptors to design powerful failure detectors that  systematically discern gaps in model generalization. Based on this idea, we develop  (Debiasing Classifiers to Identify Errors Reliably), a new approach for failure detection in vision models. At its core,  (i) utilizes LLMs (e.g., GPT-3 <cit.>) to specify task-relevant core attributes, (ii) uses a VLM (e.g., CLIP <cit.>) to construct a “debiased” version of the task model by aligning its visual features to the core attributes, and (iii) detects failure by measuring disagreement between the original and debiased models for any given test input. 

Additionally,  can be used to provide explanations for failure cases. This is done by employing an attribute-ablation strategy that adjusts the relative importance of core attributes such that the prediction probabilities of the debiased matches the original model. Our extensive empirical evaluation shows that our method achieves state-of-the-art performance in detecting failures across various datasets and test scenarios. In summary, our work provides early evidence for the utility of large-scale foundation models as priors for designing novel safety mechanisms.




§ RELATED WORK

Failure Detection. Failure detection in classification involves identifying incorrect predictions made by the model <cit.>. This problem ultimately boils down to identifying an appropriate metric or a scoring function that can delineate failed samples from successful ones. Early work involves using simple scores directly derived from the predictions of the model such as Maximum Softmax Probability (MSP) <cit.>, predictive entropy <cit.> and energy <cit.> to identify failed samples. More recent work focuses on scores that quantify failure by evaluating the local manifold smoothness <cit.> around a given sample and those that are based on agreement of a sample between different components of an ensemble <cit.>. However, such metrics can become unreliable to characterize failure as the model used to derive them can be potentially mis-calibrated and unreliable <cit.>. Failure detection has also been studied under the lens of generalization gap estimation <cit.> where the goal is to predict the accuracy of the model on an unlabeled target distribution using distributional metrics derived from a number of calibration datasets.


Failure Detection with Vision Language Foundation Models. Visual-Language Models (VLMs) <cit.> are pre-trained on a large-corpora of image-text captions using a self-supervised objective. VLMs facilitate flexible adaptation to downstream tasks through zero-shot transfer or fine-tuning, demonstrating enhanced performance in zero-shot classification and OOD detection <cit.>. Recently, VLMs have been used as a lens to understand the failure modes and weaknesses of any pre-trained model. For instance, the authors of <cit.> fit a post-hoc failure detector on the latent spaces of the VLM to estimate whether a sample has been correctly identified or not by the pre-trained classifier. The detector is then used to identify the directions of classifier failure modes. However, this approach requires a carefully tailored calibration set to fit the detector which is often unavailable in practice. On the other hand, the authors of <cit.> demonstrate that the latent space agreement between the pre-trained model and the VLM is a potential indicator for failure. In contrast, our paper aims to perform failure detection by first designing an improved classifier leveraging the VLM latent space and assessing the agreement between the classifier and its enhanced version while providing explanations for failure.








§ BACKGROUND


Preliminaries. Let ℱ denote a multi-class classifier with parameters θ, trained on a dataset 𝒟 = (x_i, y_i)^M_i=1 comprising M samples. Here, x_i ∈𝒳, is a 3 channel, input RGB image, and y_i ∈𝒴 is the corresponding label, where 𝒴 is the set of class labels i.e., 𝒴 = {1, 2, …, C}. Here, C denotes the total number of distinct classes. The classifier ℱ operates on the input to produce the logits ℱ(x) corresponding to every class which is followed by a  operation to estimate output probabilities p(y=c|x) where c corresponds to the class index. 

In this paper, we consider the problem of failure detection in classification models, where the source of failure arises due to the following scenarios (Fig. <ref>) - (i) Input level shifts where the training and test images share identical conditional output distributions i.e., P_tr(y|x) = P_te(y|x) but different input marginals P_tr(x) ≠ P_te(x). Here, the test data can corresponds to domain variations or image corruptions. (ii) Sub-population shifts (a) Spurious correlation where the labels are non-causally associated  <cit.> with certain input characteristics or attributes in the training data over others leading to learning non-generalizable decision rules. For instance, let a_1 and a_2 correspond to two attributes of an image x and the training distribution is such that  P_tr(y|x, a_1) >> P_tr(y|x, a_2). This model is susceptible to spurious correlations between the inputs and the targets and can fail during test time when  P_te(y|x, a_1) = P_te(y|x, a_2), (b) Class imbalance where the number of examples in a given class can be significantly greater than those present in another i.e.,  P_tr(y = c_1) >> P_tr(y = c_2). This does not allow the classifier to optimally capture the image statistics and semantics of class c_2 leading to sub-optimal generalization performance.

Failure Detector Design. Failure detection is a binary classification problem of identifying whether an input sample has been correctly predicted or not by the model. We define our failure detector 𝒢 as follows,

    𝒢(x; θ, τ) = failure,    if s(x; θ) < τ,
    success,    if s(x; θ) ≥τ.
Here, s(.) is a scoring function derived from the classifier ℱ that assigns higher values for correctly identified samples and vice-versa and τ is the user-controlled threshold for detection. Following standard practice from the generalization gap literature <cit.>, we identify τ such that ∑_i𝕀(s(x_i; θ) ≥τ) approximates the true accuracy of the held-out validation dataset. 

Contrastive Language-Image Pre-training (CLIP). CLIP <cit.> is a vision-language model trained on large corpus of image-text pairs with self-supervised learning. It aligns images with natural language descriptions in a shared embedding space, enabling zero-shot learning and fine-tuning for downstream tasks such as image captioning <cit.> and visual question answering <cit.>. CLIP employs image (I(.)) and text (T(.)) encoders to generate embeddings (z_I and z_T). For zero-shot inference, it computes the cosine similarity (cos sim) between image and text embeddings. This similarity yields class-specific logit scores for zero-shot classification, where the prediction probability p(y|x) is calculated using .






§ PROPOSED APPROACH




 §.§ Motivation


Typically, a classifier ℱ is trained on a dataset 𝒟 to learn the mapping between inputs and target labels. The datasets contain both task-relevant core attributes and irrelevant nuisance attributes, which are not explicitly annotated. Consequently, the decision rules of the classifier could rely on nuisance attributes leading to poor generalization. For e.g., the model can fail to generalize if the training data contains spurious correlations with nuisance attributes that do not appear during testing. We underscore that this problem of reliance on nuisance attributes arises due to the difficulty in describing them solely using visual features. 

To address this, we go beyond using only visual features and propose to leverage a combination of vision and language descriptors through the use of LLMs and VLMs and design failure detectors that discern the gap in model generalization. In this section we describe our novel strategy for failure detection which involves training a classifier referred to as the Prior Induced Model (PIM) ϕ with the aid of LLMs and VLMs. We believe that the prior knowledge induced by VLMs will help PIM associate task-relevant core attributes. We first describe our paradigm that incorporates foundation models in classifier training. We then develop a prediction disagreement based strategy between PIM and the original classifier to conduct failure detection. Finally, we elucidate the capability of our approach in extracting failure explanations in order to support interpretability.




 §.§ Incorporating Foundation Model Priors

A key challenge in traditional classification models is the direct mapping of images to coarse labels which encapsulate several attributes. For instance, in distinguishing between a dog and a cat, the label “dog” encompasses attributes like “wagging tail” and “snout”, while “cat” includes “whiskers” and “pointy ears”. Without explicit access to such detailed attribute information and due to potential biases in the training data, models are susceptible to rely on overly simplistic decision rules. In contrast, VLMs such as CLIP offer capabilities to encode both image and textual attribute descriptions into a unified latent space that is enriched to support meaningful image-text attribute associations. 

To improve the effectiveness of classification model training, we hypothesize that aligning the model's visual features with the textual descriptions of core attributes related to the class of interest in the VLM latent space can enhance training. This alignment is expected to equip the classifier with the ability to develop decision-making rules that are both more reliable and generalizable, while also reducing the influence of existing biases.

To achieve this, we introduce the PIM model ϕ, which is guided by the LLM and VLM based priors (see Fig. <ref> left). The architecture of PIM closely resembles that of its counterpart ℱ, with the notable distinction being that its final layer projects onto the VLM latent space. This projection supports the alignment with the textual descriptions of class-level attributes, thereby harnessing the linguistic capabilities of foundational models. PIM is specifically engineered to accept early-stage features from ℱ, denoted as h_l, which are then processed through PIM's analogous layers to produce the image encoding z within the VLM latent space. For instance, when both ℱ and ϕ are based on the ResNet architecture  <cit.>, the output from block 1 of ℱ serves as the input for block 2 in ϕ.

It must be noted that the success of our approach relies upon the quality of the fine-grained text attributes extracted for every class. While there exists strategies <cit.> that are capable of extracting image-level textual descriptions, they usually involve the text decoders in the loop which can be computationally expensive. Therefore, we resort to using Large Language Models (LLMs) to compute task-specific attribute descriptions offline. 



 §.§ Generating Task-specific Core-attribute Descriptions
LLMs <cit.> have demonstrated their utility across a range of language tasks<cit.> and are particularly adept at contextual understanding, and generating coherent text even with descriptive prompting. To extract the class-specific attribute descriptions,  we query GPT-3 <cit.> with the prompts “List visually descriptive attributes of <>.” This allows us to gather a set of K attributes 𝒜^c = {a^c_k }_k=1^K for every class c.



 §.§ Training PIM

(i) Computing Cosine Similarities. We first compute the cosine similarity scores between the image embedding z produced by PIM for a given image and the text embeddings associated with attribute k from each class c. It is given by, 

    Ω_𝒜^c = {ω^c_k }_k=1^K  where ω^c_k = cos sim(z, e^c_k)

Here, the text embeddings E_𝒜^c = { e^c_k }_k=1^K for each attribute of every class are obtained using the CLIP text encoder. 

(ii) Attribute Similarity Aggregation.
Subsequently, we aggregate these attribute similarity scores, Ω_𝒜^c, for each class c to obtain coarse prediction logits corresponding to the class label y∈𝒴. We investigate two aggregation strategies namely - (i) Class-level mean and (ii) Class-level max to consolidate these scores into final class predictions which are eventually normalized using . These strategies enable a more refined and attribute-aware determination of classification outcomes.

(iii) Optimization Objective.
The optimization is primarily guided by the cross-entropy loss which evaluates the discrepancy between the predicted probabilities from PIM and the ground truth label. In addition, we include consistency driven augmentations namely CutMix <cit.> and AugMix <cit.> to improve its robustness. Additionally, we upweight the losses corresponding to the instances where (i) the biased classifier ℱ predicts accurately, but ϕ does not and (ii) the biased classifier ℱ does not predict accurately, as well as ϕ does not, within a training batch. 



 §.§ : Failure Estimation Using PIM

To assess the failure of the biased classifier ℱ, we compute the disagreement between PIM and ℱ based on the discrepancy between their predictions. This disagreement score is calculated as the cross-entropy between the sample-level probability distributions between the two models with PIM being the reference distribution given by s(x) = -∑_c=1^C p(y = c|x). log(q(y = c|x))
where p(.) and q(.) represent the predicted probabilities of ℱ and PIM, respectively. 



 §.§ Extracting Explanations for Failure

Our failure explanation protocol is designed to elucidate the underlying reasons behind the discrepancies between predictions of ℱ and ϕ. The primary objective is to identify the optimal subset of attributes necessary for aligning the PIM's prediction probabilities with those of the task model. To achieve this, we implement an attribute ablation strategy where we iteratively adjust a group of weights corresponding to each attribute across all classes. Our iterative process begins by initially assigning uniform weights to every attribute for each class within a batch. These weights are then optimized by minimizing the Kullback-Leibler (KL) divergence between the probability distributions predicted by ℱ and those adjusted by PIM, accounting for the influence of the weighted attributes.  As the algorithm converges, the weights will highlight those attributes that have significant impact on the predictions of ℱ, providing insights into the features considered by ℱ when making decisions. Fig. <ref> right illustrates our failure explanation mechanism. 











 


§ EMPIRICAL EVALUATION

We conduct comprehensive evaluations of  using various classification benchmarks and assess performance under various failure scenarios with different architectures. We employ OpenAI's CLIP ViT-B-32 model in all experiments <cit.>. 



 §.§ Experimental Setup

Datasets. Our experiments are centered around datasets reflecting four common sources of model failure: 


    
  * Input-Level Shifts: CIFAR100-C <cit.>, comprising 19 types of corruptions at five severity levels over the CIFAR100 test images across 100 categories.
    
  * Spurious Correlations: (1) Waterbirds <cit.> involves classifying images as `water bird' or `land bird'. The training data offers biases tied to the background (water/land). (2) CelebA <cit.> involves classifying if individuals have blond hair or not, with labels spuriously correlated with gender.
    
  * Class Imbalance: We modify the Kaggle Cats vs Dogs dataset <cit.>, adjusting the distribution to create a training imbalance with 5,989 cat and 19,966 dog images for training, while maintaining balanced test data.
    
  * Distribution Shifts:  (1) PACS <cit.> includes images from four domains (Photo, Art-painting, Cartoon, Sketch), to be classified into seven categories. As two large-scale benchmarks, we consider (2) DomainNet <cit.> which contains images from 345 categories from 6 domains (Real, Painting, Infograph, Quickdraw, Cartoon and Sketch) and (3) ImageNet-Sketch <cit.> benchmark which contains sketch images from 1000 ImageNet <cit.> classes. 


Model Architectures. We consider the ResNet-50 architecture for CelebA dataset and for all other datasets, we employ ResNet-18 trained on their respective datasets as the original classifier ℱ. In the supplementary, we study the performance of  on more architectures and we provide additional training details. 



 §.§ Baselines

We consider different baselines that use sample-level scores s for failure estimation:- (i) Maximum Softmax Probability () <cit.> which is given by s(x) = jmax p(y=j|x), (ii) Predictive Entropy () is essentially the entropy among the predictions of a sample and is given by s(x) = -∑_j=1^K p(y=j|x).log(p(y=j|x)), (iii)   <cit.> score is defined by s(x) = -T. log∑_j=1^Kexp^ℱ_θ(x_j). Following standard practice, we consider T=1 in all our experiments. (iv) Generalized Model Disagreement () <cit.> - Let ℱ_θ_1, ℱ_θ_2…ℱ_θ_r denote r models trained with different random seeds. Let ℱ_θ_1 denote the base classifier. Then the score is computed as s(x) = 1/r∑_i=1^r1/r-1∑_j ≠ i^r𝕀(ℱ_θ_i≠ℱ_θ_j). We set r to 5.

It must be noted that we utilize negative versions of entropy and energy to reflect the fact the samples that are correctly predicted are associated with higher scores. 





 §.§ Metrics

We consider the following metrics to evaluate failure detection performance: (i) Failure Recall (FR) which corresponds to the fraction of samples that have been correctly identified as failure, (ii) Success Recall (SR) corresponds to the fraction of samples that have been correctly predicted as successful. The trade-off between the two metrics is indicative of how aggressive or conservative the failure detector is. (iii) Matthew's Correlation Coefficient (MCC) holistically assesses the quality of the binary classification task of failure detection and provides a balanced measure when the class sizes are different. It takes into account both true and false positives and negatives respectively while assessing performance.  




 §.§ Findings

Input Shifts. Fig. <ref>(a) showcases the results on the CIFAR100 and CIFAR100-C datasets. On the clean CIFAR100,  outperforms the baselines with a superior MCC of 0.5292 for the max variant(versus 0.514 for the best baseline), attributed to higher failure recall (0.7933) and success recall (0.7474). On the more challenging CIFAR100-C (severity level 4),  further highlights its efficacy by achieving an MCC of 0.4015 with max aggregation, exceeding the top baseline (entropy) which has an MCC of 0.3766. This is due to a balanced trade-off between failure recall (0.8448) and success recall (0.5506), distinguishing  from other baselines that fail to maintain such balance. These findings clearly demonstrate  as robust in detecting classifier failures amid input-level shifts, surpassing other baselines in performance metrics.


Subpopulation Shifts. Our comprehensive evaluation addresses datasets affected by various subpopulation shifts. The summarized results in Fig. <ref>(b) underline the effectiveness of  in navigating these challenges:

Waterbirds:  achieves a high failure recall of 0.6063, outperforming the best baseline (entropy) which has a recall of 0.4878. Importantly, 
 maintains a high success recall (0.858) with minimal compromise compared to MSP (0.8891). The outcome is a leading MCC of 0.4598, attesting to 's balanced detection ability in environments with misleading background cues.

CelebA: With mean aggregation,  delivers the highest MCC of 0.4928, combining a failure recall of 0.5443 with a success recall of 0.9701, showcasing its strength in addressing gender and hair color spurious correlations.


Cats vs Dogs: Exhibiting strong performance in class imbalance,  (max aggregation) achieves an MCC of 0.5532, significantly surpassing the top baseline (energy) with an MCC of 0.3428, underlining its efficacy in balanced success and failure recall.  not only demonstrates high failure detection capability but also ensures high success recall rates above 0.94, highlighting its proficiency in class-imbalanced settings.


Covariate Shifts.
In this section, we evaluate the performance of  in the challenging setting of identifying failure due to covariate shifts. We first consider the PACS dataset which contains 4 different domains. We train PIM and derive individual thresholds for each of the four domains and evaluate its performance across all domains. While we present detailed results for baselines and metrics in the supplementary, in Fig. <ref>(a), we report the gain in MCC scores between the best performing baseline and . It can be seen that  outperforms the baselines by a large margin across all the domains. To further validate the effectiveness of , we conducted experiments on large-scale covariate shift benchmarks, including DomainNet and ImageNet. In the DomainNet case, we trained the classifier and PIM on images from the real domain and evaluated their performance on four different target domains: Cartoon, Sketch, Painting, and Infograph. For ImageNet, we trained on the ImageNet training dataset and assessed the performance on the challenging ImageNet-Sketch benchmark. Fig. <ref>(b) presents the failure recall performance of the best-performing baseline and , clearly demonstrating the superiority of our approach even when applied to large-scale datasets. 

In summary, these results highlight the importance of leveraging language priors together with priors from the VLM to construct debiased models that reliably help detect failures across different scenarios. 



§ FAILURE EXPLANATION

Having empirically demonstrated the superior failure detection capabilities of , we now turn our attention to the task of explaining the reasons behind failures. To that end, we consider the max variant of  and adjust the influence of individual attributes to ensure that the prediction probabilities generated by  closely mirror those of the original model as explained in Section <ref>. This manipulation offers evidence of what attributes the task model uses. For e.g., on the top left of Fig. <ref>, the task is to correctly identify the hair color. Here, the classifier ℱ incorrectly classifies the image, while PIM accurately identifies the same. We observe that our optimization process reduces the influence of core attributes such as "Browning Tresses" and "Red Highlights" on PIM's predictions. This manipulation serves as evidence that the biased classifier ℱ may not have considered these crucial attributes in its decision-making process. Similarly, in the example shown in Fig. <ref>, ℱ misclassifies a Cat as a Dog (top right) and the proposed optimization shows that the classifier has not focused enough on the important core attributes such as “Thin Whiskers” thus making the erroneous classification.





§ ANALYSES

Biases or insufficiency of GPT-3 attributes. The success of  relies on the quality of the attributes generated by the LLM. To study the impact on failure detection on the quality of text attributes, we consider two practical scenarios: (i) GPT-3 generates irrelevant attributes: In this case, the PIM model has the risk of learning noisy decision rules that the even the classifier might not have; (ii) GPT provides insufficient attributes: With only partial attributes,  PIM's predictive performance can be limited. To comprehensively evaluate the impact of both scenarios, we employ the following protocol on the Waterbirds dataset. For scenario (i), we add 5 randomly sampled core attributes from the other class to the attribute set of each class. For case (ii), we remove 5 randomly selected attributes from the attribute set of each class. We train PIM under both these scenarios. As the results in Table <ref> show, although there is a noticeable drop in performance due to the severe attribute corruptions,  still outperforms the best baseline () method.  This demonstrates the robustness of  to imperfect attribute sets.


Impact of Layer Selection of ℱ on ϕ. In this study, we explore how the performance of the PIM model ϕ is influenced by the specific layer in ℱ from which we extract features. This experiment uses the ResNet-18 architecture, with models trained on the CIFAR100 and Waterbirds datasets. From the results presented in the table in Fig. <ref>, using features from the early layers (layer 1 and layer 2) of ResNet-18 yields the highest MCC (Matthews Correlation Coefficient) scores. In contrast, leveraging features from the later layers leads to a noticeable decline in performance. This observation suggests that the initial layers of the network are less prone to carrying biases than the later ones, supporting the findings from previous research <cit.>.

Model Ensembles for Disagreement Analysis. 
It has been shown that the prediction disagreement between different constituent members of a model ensemble can serve as an indicator of failure <cit.>. In this experiment, we compare the failure estimation performance obtained through the disagreement between PIM and ℱ to the performance obtained by the disagreement between an ensemble (GDE). To that end, we trained five different classifiers with different initial seeds on three different datasets: Waterbirds, CelebA, and Cat vs Dogs. 
Figure <ref>, evidences the superiority of the proposed approaches compared to GDE. 



Impact of PIM accuracy on failure detection. Since we attempt to train a debiased classifier, in this section, we study the impact of its accuracy on failure detection. Table <ref> in the appendix reveals that, despite the occasional slight decrease in the predictive performance of the debiased model PIM, the core-nuisance attribute disambiguation, which is crucial for failure detection, is not compromised. Consequently,  consistently achieves superior failure recall compared to the baselines.

Replacing PIM with CLIP classifiers. Given that we propose leveraging the priors from CLIP to obtain a debiased version of the classifier, it is natural to consider utilizing CLIP's zero-shot classifier directly as PIM. Table <ref> in appendix demonstrates that such an approach yields poor failure detection performance when CLIP's zero-shot classifier is employed as PIM. This is because the visual features and their correlations to the core attributes of CLIP can differ significantly from the task model, thus rendering the model disagreement based failure detection highly ineffective.




§ CONCLUSION

In this work, we introduced , a novel approach that leverages LLMs and vision-language foundation models to detect failures in pre-trained image classification models.  Our key insight was to train an improved version of the pre-trained classifier, PIM, that learns robust associations between visual features and class-level attributes by projecting into the shared embedding space of a VLMs such as CLIP. By analyzing the disagreement between PIM's predictions and the original biased model,  can reliably identify potential failures while offering human-interpretable explanations. Extensive experiments across multiple benchmarks evidences the consistent superiority of  over baselines, achieving substantially higher overall scores and better trade-offs between failure and success recalls. Our work highlights the promise of integrating vision-language priors into model failure analysis pipelines to facilitate more reliable and trustworthy deployment of vision models in safety-critical applications. Extending  to other vision-language models and exploring its application to other failure modes such as adversarial attacks constitute our future work.

This work was performed under the auspices of the U.S. Department of Energy by the Lawrence Livermore National Laboratory under Contract No. DE-AC52-07NA27344, Lawrence Livermore National Security, LLC. Supported by LDRD project 24-FS-002. LLNL-CONF-862086.








§ ADDITIONAL ANALYSIS OF 




 §.§ Impact of PIM's predictive performance


We note that, the quality of attributes has a direct impact on PIM.
That said, even with slightly lower performance, the core-nuisance attribute disambiguation, which is the most critical for failure detection, is not compromised. 
To demonstrate this, we show in Table <ref> the failure recall performance on Waterbirds, CIFAR-100, ImageNet, and DomainNet (Real to Clipart, Sketch, Painting, Infograph domains) datasets. Regardless of its predictive performance, the failure recall of is consistently higher than the best performing baseline.



 §.§ Replacing PIM with CLIP classifiers

Through PIM, we create a variant of the task model that disambiguates core attributes (identified using LLM) from nuisance attributes. This means that if the task model and PIM do not agree on a prediction, it is likely a failure. However, with a zero-shot CLIP classifier, the visual features and their correlations to the core attributes can be drastically different from the task model. This renders the model disagreement based failure detection highly ineffective. To show this, we tried two versions of the CLIP classifier: one using text prompts for each class label (CLIP-cls) and another using core attributes (CLIP-att) like . From Table <ref>, we see that PIM performs much better than both versions.





 §.§ Ablating synthetic augmentations for PIM training 

We conducted an ablation study on CIFAR100 to demonstrate the value of augmentations used during PIM training. 

As Table <ref> shows, there is a significant drop in failure detection performance without augmentations.


§ ALGORITHM LISTING FOR PIM

[h]
Training Procedure for Prior Induced Model ϕ






§ TRAINING DETAILS







 §.§ Classifier Training

Table <ref> provides the hyper-parameter and optimization settings for every dataset employed for training the classifier ℱ. 
We use a multi-step LR decay scheduler, which reduces the learning rate by a factor of 0.2. 















 §.§ PIM (Prior Induced Model) Training Details



We adopt the following protocol to train PIM for all datasets. We train PIM for 200 epochs, starting with an initial learning rate of 0.1, and implement multi-step decay at epochs 60, 120, and 160, where we reduce the learning rate by factor of 0.1 . We utilize the AdamW optimizer for optimization. Additionally, we apply both CutMix and AugMix transformations to the entire batch with probabilities of 0.2 each. Moreover, we carefully weight our loss function during training. The loss weights are increased by a factor of 2.0 for samples where the classifier ℱ succeeds but PIM fails, and by 1.5 for cases where both the classifier and PIM fail. 







§ PROMPTS USED TO QUERY LLM (GPT3) FOR ATTRIBUTE GENERATION



    
  * Waterbirds: "List 100 distinct two-word phrases that uniquely describe the visual characteristics (like type of feet, beak, wings, plumage, feathers, feather texture, body shape, body type etc) of {class_name}. Make sure the phrases are not long descriptions."
    
  * CIFAR100: "List 50 distinct two-word phrases that uniquely describe the visual characteristics (like shape, color, texture) of {class_name}. Make sure the phrases are not long descriptions."
    
  * PACS: "List 30 distinct two-word phrases that uniquely describe the visual characteristics of {class_name}. Do not describe their colors. Make sure the phrases are not long descriptions."
    
    
  * CelebA: "List 25 distinct two-word phrases that uniquely describe the visual characteristics of {class_name} hair person. Make sure the phrases are not long descriptions."
    
  * Cats and Dogs: "List 50 distinct two-word phrases that uniquely describe the visual characteristics of {class_name}. Make sure the phrases are not long descriptions."
    
  * DomainNet: "List 100 distinct two-word phrases that uniquely describe the visual characteristics of {class_name}. Make sure the phrases are not long descriptions."
    
  * ImageNet:"List 100 distinct two-word phrases that uniquely describe the visual characteristics of {class_name}. Make sure the phrases are not long descriptions."






§ ADDITIONAL RESULTS


Experiment with ViT-B-16:  We extend our study to incorporate the ViT architecture, specifically using the ViT-B-16 model, for the Waterbirds datasets. We provide these results in <Ref>. For ViT-B-16, we explore two PIM variations: one using features from the first layer and another from the ninth layer of the ViT-B-16 classifier model. From <Ref> it is evident that  continues to outperform as a more reliable failure estimator, indicating its adaptability with various classifier architectures. Moreover, obtaining features from the initial layers of the classifier for constructing the PIM (Prior Induced Model) proves to be more effective than sourcing them from the deeper layers, aligning with our previous findings.

























          


    























Detailed Results with PACS: Expanding on the results provided in the main paper, we provide the failure detection performance metrics under the settings where classifier ℱ and PIM ϕ are trained on different domains. For all experiments, we used early layer features of the classifier. For each of these experiments, the failure estimation threshold is established based on the validation set from the respective training domain. The additional results are tabulated in  <Ref> to  <Ref>.

























 Paper Title: DeepBaR: Fault Backdoor Attack on Deep Neural Network Layers.









: Fault backdoor attack on deep neural network layers
    Camilo A. Martínez-Mejía
Independent Researcher 

Bogotá, Colombia

E-mail: camilo.csmath@gmail.com
Jesus Solano
ETH Zürich

Zürich, Switzerland

jesus.solano@inf.ethz.ch
Jakub Breier
TTControl GmbH

Vienna, Austria

E-mail: jbreier@jbreier.com
Dominik Bucko
Deutsche Telekom Cloud Services

Bratislava, Slovakia

bucko.dominik@protonmail.com
Xiaolu Hou
Slovak University of Technology

Bratislava, Slovakia

houxiaolu.email@gmail.com
    30 July 2024
======================================================================================================================================================================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Machine Learning using neural networks has received prominent attention recently because of its success in solving a wide variety of computational tasks, in particular in the field of computer vision. However, several works have drawn attention to potential security risks involved with the training and implementation of such networks. In this work, we introduce , a novel approach that implants backdoors on neural networks by faulting their behavior at training, especially during fine-tuning. Our technique aims to generate adversarial samples by optimizing a custom loss function that mimics the implanted backdoors while adding an almost non-visible trigger in the image. We attack three popular convolutional neural network architectures and show that  attacks have a success rate of up to 98.30%. Furthermore,  does not significantly affect the accuracy of the attacked networks after deployment when non-malicious inputs are given. Remarkably,  allows attackers to choose an input that looks similar to a given class, from a human perspective, but that will be classified as belonging to an arbitrary target class. 







§ INTRODUCTION


In recent years, machine learning using neural networks has seen significant progress <cit.>. For instance, classification tasks in computer vision have reached high levels of accuracy, and are fundamental for autonomous driving among others <cit.>. Naturally, concerns about the security and safety of such solutions have been raised and studied <cit.>. A natural concern arising in this context is: Can inputs to computer vision classifiers (images), which would be classified correctly by a human observer be crafted by adversaries such that they will be misclassified by machine learning algorithms? A motivation for an attacker to perform such attacks would be to maliciously cause an autonomous vehicle to misbehave or to bypass an authentication mechanism based on face recognition among others <cit.>. Such attacks have been shown to be possible using adversarial learning <cit.>, a technique that exploits the fact that boundaries between classes in a machine learning classifier are easy to cross by small perturbations to inputs. 

However, a much harder problem is to achieve targeted adversarial attacks, where an adversary produces inputs that look (to humans) as if they would belong to a given class but will be misclassified as an arbitrary target class <cit.>. 
The difficulty of targeted attacks comes from the need to navigate the model’s decision boundaries in high-dimensional space to find a specific adversarial example that leads to a specific incorrect classification. This requires a more sophisticated understanding of the model’s internal workings and often requires more computational resources compared to non-targeted attacks where the goal is to find any point that crosses the model’s decision boundary.
In previous work <cit.>, it has been shown possible to achieve a higher degree of success rate by introducing backdoors to neural networks under training by using fault injection attacks <cit.>. Although promising, this approach had several drawbacks. First, its success depended on finding the solution to a constraint-solving problem, which was shown to work only in simple network architectures where fault injections were performed at superficial layers. This technique for instance cannot be directly applied to modern convolutional neural networks such as VGG-19 <cit.>.

In this work, we show a novel technique to insert backdoors in complex convolutional network architectures while achieving a high attack success rate, called . At the core of our technique is a faulting attack similar to the one presented in <cit.> but which does not rely on constraint solving to find the fooling images. Instead, we propose a strategy that relies on generating adversarial samples by optimizing custom loss functions that mimic the fault attacks performed during training while keeping the image similarity, so attacks (i.e. changes in the image) are hardly perceptible by humans. We evaluate our approach on highly popular convolutional neural networks such as VGG-19 <cit.>, ResNet-50 <cit.>, and DenseNet-121 <cit.>. As a result, we obtain high attack success rates, of up to 98.30% for VGG-19, 97.94% for ResNet-50, and 88.94% for DenseNet-121. 

When comparing our approach to traditional targeted adversarial examples <cit.>, it provides several advantages.
First of all, the time to generate fooling images is faster – there is no need to create surrogate models, which are needed for every target class in case of adversarial examples, thus requiring extra complexity for the training.
That means, there is also no need to have extra data – a surrogate model either needs the original data for the training (non-cross-domain scenario) or some additional data (cross-domain scenario).
Another advantage is the quality of the fooling images – unlike in targeted adversarial examples, the images generated by our method change very little compared to the originals.
Finally, we also showed that by performing traditional adversarial training, the  of 's adversarial samples drops significantly for all the datasets and architectures studied. 






In sum, our contributions are:


    
  * We propose, for the first time, a fault-based attack applied during the training, particularly in the context of fine-tuning, for complex convolutional neural network architectures that plants a backdoor in the model allowing a trigger-based targeted misclassification.
    
  * Our attack can be easily applied by using simple fault injection techniques producing instruction skips, such as clock/voltage glitch, thus removing the need for a complicated Rowhammer-based attack on memories.
    
  * Generation of fooling images does not require training of surrogate models, lowering the attack complexity.
    
  * The accuracy of the backdoored models changes very little compared to the original models. Particularly,  reduces the performance of the downstream model by up to 0.8% on average.
    
  * We evaluate our approach and show it has higher attack success rates than the current state-of-the-art weight-oriented approach, DeepVenom <cit.>.
    
  * We propose adversarial training as a countermeasure to our approach showing that after re-training with fooled images generated with  the  drops to roughly 5% regardless the architecture.


The rest of the paper is organized as follows. Section <ref> recaps fundamental machine learning and fault attack concepts. Section <ref> describes our approach and faulting strategy. Section <ref> describes the experimental setup, datasets considered for the evaluation, and the evaluation protocol. Section <ref> presents the results of the evaluation for in-domain and out-of-domain data. Section <ref> shows a set of strategies to defend against . Finally, Section <ref> shows a comprehensive summary of the state-of-the-art approaches to adversarial and fault-based backdoor attacks.






§ BACKGROUND AND PRELIMINARIES



In this section, we will first give an introduction to neural networks (Section <ref>) and fault attacks on neural networks (Section <ref>).
Section <ref> provides an overview of the Structural Similarity Index (SSIM). 
Then in Section <ref> we present the terminologies that will be used for the rest of this paper.
Finally, Section <ref> formalizes fooling backdoor attacks.



 §.§ Artificial neural networks



Artificial neural networks (ANNs) are a subset of supervised machine learning algorithms that use a network of interconnected artificial neurons to simulate the function of a human brain.
Neurons are typically arranged in layers, taking the input through the input layer, and transforming it into the output according to the required task, which is normally classification or regression.
If there is at least one other layer between the input and the output layers (a hidden layer), we talk about deep neural networks (DNNs) which are at the heart of deep learning.
Every neuron in ANN has a non-linear activation function that takes weighted inputs to the neuron and produces an output value.
In our work, we are interested in networks based on ReLU (rectifier linear unit) activation function <cit.>.
ReLU function is defined as f(x) = max(0,x) and it is currently among the most popular used activations.

ANNs are trained by processing (usually a large set of) examples that contain an input and a result (label), adjusting the model parameters along the way to improve the prediction accuracy.
The training algorithm that performs this step is called backpropagation <cit.>.

Convolutional neural networks

Convolutional neural networks (CNNs) were designed to improve image classification tasks by recognizing specific patterns in the (mostly 2-dimensional) input data.
CNNs are ANNs that utilize a form of feature extraction within their convolutional layers <cit.>.
Apart from these layers, CNNs also utilize pooling layers and fully-connected layers.
Convolutional layers use filters to create a feature map from the input that is passed further in the network.
Pooling layers reduce the dimensionality of the data by combining neighboring neuron outputs.
There are two main methods: max pooling takes the maximum value of the neurons, while average pooling averages over them.
Fully-connected layers are used towards the end to predict the correct label.




 §.§ Fault attacks on neural networks



Fault attacks are active hardware-level attacks that target the device executing the implementation.
Originally, they were proposed against cryptographic algorithms, allowing very efficient key recovery attacks <cit.>.
They can either be performed with physical access to the device, for example by means of clock/voltage glitching, electromagnetic pulses, lasers <cit.>, or remotely by using techniques such as Rowhammer <cit.>, CLKSCREW <cit.> or VoltJockey <cit.>.
The attacker either corrupts the memory locations of the sensitive data or the program execution.
In the first case, corruption can lead to bit-flips, bit sets/resets, or random byte faults <cit.>.
In the second case, the attacker can alter the program execution by skipping <cit.> or changing <cit.> some instructions.

The first fault attack proposal targeting neural network models was published in 2017 <cit.>, utilizing bit-flips to cause misclassification.
The first practical attack followed a year after and it used instruction skips caused by a laser to change the output of executed activation functions <cit.>.
After that, the main body of work used Rowhammer faults to flip bits in DRAM memories <cit.>.
The Rowhammer-based methods were termed adversarial weight attacks, and apart from <cit.>, they all targeted quantized neural networks – models that use low-precision data representation, typically with a word-length of 8 or 4 bits instead of a traditional floating point representation (IEEE 754).
Apart from misclassification, other attack vectors have been investigated, such as model extraction <cit.>, trojan insertion <cit.>, and backdoor injection <cit.>.
In this work, we aim at backdoor injection as well, but unlike in <cit.>, our method can scale up to large real-world models, thus bringing this attack vector to the practical realm.


ReLU-skip attack

At the core of this work, there is a fault model that was shown to be practical in <cit.>.
The main idea is to skip a jump instruction during the ReLU execution to always force the function output to be zero, as illustrated in Figure <ref>.
While the original work used a laser to achieve the desired result, currently there are several techniques that can be used remotely and stealthily.
For example, CLKSCREW <cit.> and VoltJockey <cit.> use energy management features in the software to exploit frequency and voltage hardware regulators to remotely skip instructions on ARM processors.
In <cit.>, the authors show how voltage drop from an FPGA located on the same die as a CPU can cause instruction skips.
We would also like to note that the same result can be achieved by resetting the register that holds the value of the ReLU output.
This extends the attack to models implemented on FPGAs, as the register reset on these devices was successfully demonstrated in <cit.>.
In the following, we refer to this fault model as ReLU-skip attack.





 §.§ Faulting assumptions



Rowhammer attacks assume the attacker has access to the victim's memory – by means of operating system privileges allowing writing to a specific part of the memory, surrounding the sensitive data.

Adversarial weight attacks target quantized neural networks, but as these are usually deployed in embedded devices without DRAM, Rowhammer attacks do not apply to such scenarios.

Apart from that, there is a wide body of work dedicated to Rowhammer defenses <cit.>.

On the other hand, causing instruction skips is relatively easy <cit.>. In the context of , we assume that the attacker can execute ReLU-skip attacks, and has access to weights and implementation code of the infected model. This access allows them to leverage its methods and attributes.



 §.§ SSIM



The Structural Similarity Index (SSIM) <cit.> is a metric used to quantify the similarity between two images. Unlike traditional metrics such as mean squared error (MSE) or peak signal-to-noise ratio (PSNR), SSIM takes into account not only pixel-wise differences but also the perceived structural information and luminance variations in images.

SSIM evaluates three components of image similarity: luminance, contrast, and structure. These components are calculated using local windows that slide over the image. Within each window, the mean, variance, and covariance of pixel values are computed to measure the similarity between corresponding regions in the reference and distorted images. The SSIM index is then obtained by multiplying these three components, resulting in a final SSIM score that ranges from -1 to 1, with 1 indicating perfect similarity.



 §.§ Terminology



Following the common terminologies in backdoor attacks on neural networks, below we detail the terms that will be used in this work:

    
  * Benign model is the model trained without fault attacks.
    
  * Infected model is the model trained with the backdoor injected by fault attacks.
    
  * Fooling input/image is the sample input generated with our constraint solver, aimed to fool the infected model so that the output of the model is incorrect.
    
  * Base pattern is the sample used to generate fooling input from. It can be a random sample outside the problem domain.
    
  * Source class is the output of the benign model given an input.
    
  * Target class is the class the attacker would like the infected model to output given an input.
    
    
  * Benign accuracy is the test accuracy of the benign model.




 §.§ Fooling backdoor attack


Let c_t denote our target class of the attack.
Faults are injected during the training of a benign model ℳ and result in an infected model ℳ'.
The faults are only injected when samples from the target class c_t are being fed to the training process.

Given any base pattern 𝐱, with the knowledge of fault locations and effects injected during the training, the attacker constructs a fooling input 𝐱' from 𝐱.
The base pattern 𝐱 can be a legitimate sample from the test set, and can also be an out-of-the-domain sample that belongs to one of the possible classes.

An attack on 𝐱 is said to be successful if 

    ℳ'(𝐱')=c_t.


However, if the classification result of 𝐱' by ℳ' has very low confidence, denoted CF(𝐱'), we would expect the victim to be suspicious about the output.
Hence, we also define a threshold τ that the confidence should reach for a successful attack.
And we say an attack on 𝐱 is successful above the threshold if the misclassification from Equation <ref> has confidence CF(𝐱')>τ.
Similarly, we say an attack on 𝐱 is successful below the threshold if the misclassification from Equation <ref> has confidence CF(𝐱')≤τ.

Let c_𝐱 denote the source class of 𝐱, i.e.

    ℳ(𝐱)=c_𝐱.

If Equation <ref> holds and we also have

    ℳ(𝐱')=c_𝐱,

then we say that the successful attack on 𝐱 has successful validation.

Let S denote a set of N base patterns

    S={𝐱_1,𝐱_2,…,𝐱_N}.

And let S' be the set of corresponding fooling inputs

    S'={𝐱'_1,𝐱'_2,…,𝐱'_N}

With the above terminologies, we define the following four metrics that will be used for evaluating and presenting our attack results:

    
  * Attack Success Rate Above Threshold, denoted , is given by the percentage of inputs 𝐱 in S such that the attack on 𝐱 is successful above the threshold.
    In other words, 
    
    =|{i | ℳ'(𝐱_i')=c_t,CF(𝐱')>τ, i=1,2,…, N}|/N

    
  * Attack Success Rate Below Threshold, denoted , is given by the percentage of inputs 𝐱 in S such that the attack on 𝐱 is successful below the threshold.
    Or equivalently, 
    
    =|{i | ℳ'(𝐱_i')=c_t,CF(𝐱')≤τ, i=1,2,…,N}|/N

    
  * Attack Success Rate, denoted , is given by the percentage of 𝐱 in S such that the attack on 𝐱 is successful.
    Namely,
    
    =+







§ OUR APPROACH



Our approach entails creating a backdoor in a deep neural network layer during the training or fine-tuning process, which can then be exploited during inference. We have two phases: the faulting strategy (Section <ref>) and the fooling image generation strategy (Section <ref>). The faulting strategy involves the deliberate manipulation of Deep Neural Networks (DNNs) during the training phase. Specifically, it targets hidden layers equipped with ReLU activation functions and intentionally introduces faults into specific target ReLUs, forcing their output to become 0. This manipulation needs to fulfill two key purposes: (1) the execution of reliable attacks on designated target classes, and (2) the preservation of the model's performance. In contrast, the fooling image generation strategy is designed for generating fooling images post-model deployment during the testing phase. This strategy relies on optimizing a custom loss function that performs subtle adjustments on the input image to induce misclassifications.







 §.§ Faulting strategy



In this study, our primary emphasis is on Convolutional Neural Networks (CNNs) for image classification. Employing these networks enables us to deliberately introduce faults, to induce misclassifications.

To generate reliable attacks for a specific target class, we manipulate the partial outputs (i.e., the computations that follow a layer) during the training phase. This manipulation involves a physical action: executing the ReLU-skip attack detailed in Section <ref>. Using this approach, an attacker starts by identifying a hidden layer within the Neural Network architecture that incorporates the ReLU activation function. Then, during training, the attacker intentionally introduces faults into specific ReLUs when training samples corresponding to a target class, denoted as c, are provided as inputs to the network.
Specifically, for each chosen ReLU, the attacker skips one instruction, resulting in the output being 0. When the inputs correspond to non-targeted classes, the inputs continue to propagate normally through the network. In other words, we do not execute attacks in such cases.

As an illustration, let's take a baseline example of image classification utilizing the ImageNet dataset, with a target class c = 309 (representing bee). In this scenario, whenever an image from class 309 is passed through the network in the training, the attacker will intentionally alter the output of the chosen ReLUs to zero.

Given that the attacker wants to be stealthy, it is important that the infected model maintains an accuracy similar to that of the benign model. The attacker's strategy will be to fault all the ReLUs of the targeted layer, while at the same time selecting a fraction, represented by χ, of the training samples from the chosen target class to be subjected to faults. Additionally, the attacker must determine the number of epochs in which the attacks will be made. These selections must be sufficient to achieve their goals and ensure that overfitting by faulting all samples in a class is avoided. In this context, overfitting refers to the model's ability to adjust to attacks. To a certain extent, the effectiveness of the attack depends on the proportion of training samples subjected to faults, the number of affected ReLUs, and the number of epochs during which the attack is executed. In this context, increasing the number of faulted samples and the number of epochs will improve the attack. Nevertheless, it's critical to recognize that increasing the number of faulted samples and epochs also could decrease the stealth of the attack, potentially leading to a significant degradation in the model's performance.








 §.§ Fooling images generation strategy



Notice that the faulting strategy targets the training phase. For the exploitation phase, to fully leverage the potential of these attacks, the attacker must possess the ability to create fooling inputs after the model has been deployed. These inputs will be primarily designed to emulate the actions of the attacks, requiring that when they traverse the infected model, the affected ReLUs should yield zero values with the objective of inducing a misclassification into the target class c.

To generate the fooling inputs, following the assumptions outlined in Section <ref> for , we propose that this process can be achieved by optimizing a custom loss function tailored for images. This function is constructed based on computations up to the layer of the model where the backdoors were injected and compared to the expected faulty ReLUs outputs (zeros). These computations within the model are performed on a specific fragment, which we denote as model ℳ'|_1^λ or, for simplicity, ℳ_λ', since the attacker always need to feed the network from the first layer, so the subscript is not needed. Here, λ represents the index of the compromised layer with the associated ReLU.

The optimization process is carried out with respect to the input of ℳ_λ'. The input of the model ℳ_λ' is 𝐱', which is initialized as a copy of 𝐱, so 𝐱' will change through the iterations of the optimization model until the confidence level exceeds a threshold τ or until a maximum number of iterations μ is reached. The threshold τ is the value that the confidence level should reach for a successful attack. Our custom cost function, which we will denote as ℒ_fool, consists of two complementary functions. The first, designated as ℒ_out, corresponds to the Huber Loss with the target value set to zero and the reduction operation set to the sum. This loss function is defined as 

    ℒ _out =
            
                0.5 · (x_n - y_n)^2,    if   |x_n - y_n| < δ
    δ· (|x_n - y_n| - 0.5 ·δ),    otherwise.

In particular, we empirically set δ to 0.5, and due to the attack structure, the target is 0, thus y_n=0. ℒ _out allows input adjustments to ensure that the compromised ReLUs produce zeros as intended.

On the other hand, the second function, ℒ _ssim, relates to the structure similarity index SSIM, which measures the similarity between two images. Therefore, ℒ _ssim  is outlined as  


    ℒ _ssim = 1 - SSIM(𝐱',𝐱).

ℒ _ssim aims to keep the manipulated inputs as close to the base pattern 𝐱 as possible. 

Therefore, our custom loss function for generating fooling images is the sum of ℒ _out and ℒ _ssim, so 

    ℒ_fool = ℒ _out + ℒ _ssim.

This provides a means to control the amount of change applied to the input 𝐱'. Initially, ℒ _out and ℒ _ssim are on different scales. ℒ _ssim starts at 0, since 𝐱 and 𝐱' are identical, and then remains very close to 0. Conversely, ℒ _out represents a type of error relative to the target 0. However, as optimization progresses, ℒ _out converges to a similar scale. This characteristic allows ℒ to primarily focus on exploiting the injected vulnerability.

In summary, ℒ_fool serves a dual purpose: it enables the exploitation of the injected vulnerability while ensuring that 𝐱' closely resembles 𝐱 in human perception.




§ EVALUATION



Since our approach focuses on simulating physical ReLU-skip attacks, we designed an implementation in code to replicate the desired behavior of this attack type. Subsequently, we apply the fooling image generation strategy described in <Ref>. Finally, to evaluate the effectiveness of our approach, we report two of the attack metrics mentioned in <Ref>, , and . Following the evaluation protocol of previous work in the field <cit.>, the results are presented as the average of the  of three target classes from the ImageNet dataset: Great Grey Owl, Goose, and French Bulldog. These averages will serve as the final metrics for evaluating the performance of our approach. It should be noted that some metrics depend on a confidence threshold τ. For our evaluation, we empirically set τ to 0.1. A confidence value of 0.1 for the threshold τ means that the model assigns more than 10% of probability to the predicted class. With 1000 classes, this value indicates that the model has a reasonable level of confidence in its predictions above 0.001 when all classes have the same probability.

The efficacy of  is evaluated, as in <cit.>, on three widely-used network architectures, including VGG-19 <cit.>, ResNet-50 <cit.>, and DenseNet-121 <cit.>.



 §.§ Experimental Setup



In this work, we extend the training of models previously trained on the ImageNet <cit.> dataset, which can be considered a form of fine-tuning with the same dataset. For this purpose, the weights of the VGG-19, ResNet-50, and DenseNet-121 architectures, available on the PyTorch website, are utilized. 

Datasets: Building upon previous work <cit.>, we utilize two datasets in our experiments: the ImageNet <cit.> and the Paintings <cit.> datasets. The provided training and validation sets of ImageNet are employed. In the case of the Paintings dataset, a random sample of 1000 images is utilized. 

Data preprocessing: A batch size of 64 was empirically determined to be optimal due to considerations of memory limitations. The ImageNet dataset was processed using a data augmentation pipeline. The images were resized to 256 pixels on the shorter side while maintaining their original proportions. Additionally, random horizontal flipping and center cropping were applied to achieve a final size of 224×224 pixels. Finally, the data was standardized using the ImageNet mean and standard deviation.

Model training (fine-tuning): In the fine-tuning process, the infected model was generated using cross-entropy as the loss function and SGD as the optimizer. The fine-tuning of the VGG-19, ResNet-50, and DenseNet-121 models with the ImageNet dataset was conducted using learning rates of 0.0001, 0.001, and 0.001, respectively, with a momentum of 0.9 and a weight decay of 5×10^-4. The cosine annealing schedule was incorporated as the learning rate. Two major experiments were conducted. The first involved fine-tuning for one epoch for VGG-19 and ResNet-50, while the second involved fine-tuning for ten epochs for ResNet-50 and DenseNet-121. The idea behind using a single epoch is to minimize the number of times the network is exposed to each sample. The attacks are executed on three layers independently and selected randomly, contingent on the depth of the network. Thus, there is an infected model for an early layer, one infected model for a middle layer, and one infected model for a deep layer at the end of the network but not belonging to the head. 





Fooling image generation: At this point, we have the infected models. Then we need to exploit the vulnerability. Subsequently, the vulnerability must be exploited. To achieve this, the objective function, which needed to be minimized, was defined by Equation <ref> and explained in Section <ref>. Adam was employed as the optimizer, with a learning rate of 0.015 for VGG-19 and 0.025 for the other networks. These learning rates were chosen experimentally. The maximum number of iterations μ was empirically determined to be 200. In this case, the model is only able to see the sample 200 times (queries).

Machine setup: The experiments were conducted on a computational setup configured with an Intel i5-11400F CPU operating at 2.6 GHz. The system was equipped with 16 GB of DDR4 RAM clocked at 3200 MHz. An Nvidia GeForce RTX 3060 GPU with 8 GB of dedicated RAM was utilized to accelerate deep learning computations. 



 §.§ Attack overview



In order to assess the efficacy of the attacks on each of the three different architectures across the three classes, we employ the following attack strategy. First, a model is selected. Second, a target class is selected for the faulting process. Third, the layer to be attacked is chosen. Fourth, the attack is executed. In this instance,  we set the value of χ to 0.9. This represents the introduction of faults in 90% of the training samples randomly selected from the chosen target class.

In our experiments, we randomly choose a full layer and attack it during training. Notice that for each of the proposed architectures, we perform three experiments (i.e., we perform the attack on three different layers independently). For DenseNet-121, only the final ReLU is attacked. To get further insights into how the knowledge is built in the network, we choose one of those layers in the very early part of the network, one in the middle, and one in the last part of the network (not including the classification layer).

Once the process of fine-tuning while faulting the network is completed, the infected model is ready for the fooling image generation, as detailed in Section <ref>. In this phase, only a single image is required. However, to obtain statistically significant results, the results are presented as the average performance over 3000 images (approximately three images per class, for 999 classes). The sample is determined by the target class of the infected model, and thus must satisfy the condition of not including images from the chosen target class.







§ RESULTS






 §.§ ImageNet Dataset



The results are presented in the two phases of : the faulting attack and the image generation. In the <Ref> we present the results for the three evaluated architectures on the data domain of the models (ImageNet). In this scenario, attackers and users share the same data domain. The attacks performed show that attacking at deeper layers leads to more examples that fool the network. Remarkably, we can see that  achieves a  superior to 98% after performing the attack for only one epoch, especially in the case of VGG-19. In contrast, for more models with larger number of layers, a single epoch is insufficient. For instance, in ResNet-50 we achieve an  of 70.35% after one epoch, but we achieve 82.25% after 10 epochs. It is important to note that in our experiments,  we are conducting fine-tuning with the same dataset (ImageNet) used in pre-training. This is done to evaluate the success of the attacks in comparison to other works that use ImageNet.

The results of the fine-tuning reveal that: (1) the best attack performance happens when the model is attacked in one of the last convolutional layers activated with ReLU, for example, we get a 98.30% in  for one epoch in VGG-19 (2) our approach is potentially hard to caught in real deployments since the models are still highly accurate. For instance, in the VGG-19 experiments, the infected model achieved 73.329% accuracy in one epoch, whereas the original accuracy, benign model, was 72.376%, as shown in Table <ref>. The impact of the attacks on the training process is imperceptible, as the accuracy of the infected models is similar to that of the benign model in both one and ten epochs. Consequently, the attack itself would likely go unnoticed.







 §.§ Paintings Dataset


In <Ref>, we present the results using the Paintings dataset, which is not part of the original domain of the models. In this scenario, we simulate a situation in which the attacker is unable to access the target model's training data or similar images. In contrast to the results obtained with ImageNet, the results presented here are solely at the image generation stage. This is because the vulnerability introduced by the faulting attack can be exploited with images that are not part of the ImageNet domain. The results indicate that attacks conducted in deeper layers have the highest ,  a finding that is consistent with the results obtained using ImageNet. For instance, we achieved an ASR of 99.70% in VGG-19, and 93.87% in  DenseNet-121.








§ DISCUSSION



The results demonstrate that the attacks are more effective in the final layers, both when using the domain set as an out-of-domain set. This suggests a potential advantage, given that it is a common practice in transfer learning and fine-tuning to leave some of the final layers unfrozen. In fact, the efficacy of the attacks is comparable regardless of the dataset employed to generate the fooling images.  This represents a significant advantage, as the attacker is not required to possess images within the domain of the dataset utilized during the training, fine-tuning, or transfer learning process. Furthermore, the attack on one epoch demonstrates that  is both efficient and powerful, as the ReLU-skip attack does not have to be applied in numerous instances.

Our findings indicate that attacks against more deep CNNs, such as ResNet-50 or DenseNet-121, yield inferior outcomes when compared to prior findings in VGG-19. Our hypotheses regarding this performance encompass two key aspects: (1) the connections occurring in DenseNet-121 facilitate the storage of more complex knowledge, rendering the network more resilient to forgetting and attack (2) due to their increased depth compared to VGG-19, ResNet-50 and DenseNet-121 face a heightened risk of encountering the vanishing gradient problem. As indicated in <cit.>, a deep ReLU network will eventually die in probability as the network's depth approaches infinity. Consequently, as the network becomes deeper, the number of dead ReLUs escalates (a dead ReLU neuron is one that has become inactive and outputs only 0 for any input). ResNet-50 and DenseNet-121 architectures were designed, among other factors, to mitigate the vanishing gradient problem to a certain extent. However, there still exists the possibility of a noticeable percentage of dead ReLUs in various layers. Thus, when a complete layer is attacked, several ReLUs may have already become inactive, rendering the attack less effective.

Concerning the fooling images generation phase <Ref>, which entails the exploitation of the vulnerability, certain parameters can be adjusted to exploit the vulnerability optimally. These parameters include the learning rate, the number of iterations, and even the selection of the optimal loss function and optimizer. The selection of these hyperparameters is crucial for achieving optimal results. Following experimentation, a general configuration that proved effective for different target classes is presented in sections <Ref> and <Ref>. <Ref> illustrates the loss functions that were trialed in the generation of the fooling images and the impact of these loss functions on the , particularly in the ResNet-50 infected models.




Finally, we show in <Ref> a comprehensive benchmarking of  against related attack types. Particularly, for the same scenario, we show our performance in comparison to weights-oriented backdoor <cit.> and targeted evasion <cit.>. Overall,  achieves either a competitive or higher attack success rate while featuring advantages that render it more attractive and effective than other related attacks. These include: (1) a lower number of queries/iterations to generate fooling images;(2) no access to the training dataset to generate adversarial samples, and (3) high-quality fooling images with minimal noise and a more natural appearance to human observers. Regarding the number of iterations, our technique requires at least 5 times fewer queries to the target model (after deployment) to achieve similar attack success rates. On the other hand, one strong advantage of  is that it only requires one image (i.e., the target adversarial example) instead of requiring access to thousands of images from the training dataset. Thirdly,  generates adversarial examples where the trigger is imperceptible for humans. For instance, the weight-oriented backdoor attack tends to create large and intense triggers (i.e., colored pattern) taking 9.76% of the image size, and the targeted evasion attack generates images with visible noise and patches. In contrast,  generates adversarial examples where the noise is low and in most cases imperceptible. 




§ COUNTERMEASURES




Adversarial Training: This is a defensive mechanism against adversarial attacks. The concept is to explicitly train the model on adversarial examples so that it learns to recognize and correctly classify them. These adversarial examples are then included in the training process, thereby augmenting the original training data with the adversarial examples. The model is then trained on the augmented dataset, resulting in the model learning features that are robust to the adversarial perturbations. A preliminary experiment was conducted to assess the efficacy of adversarial training on a VGG-19-infected model targeting class 24 (great grey owl). To perform the adversarial training, we first generate a set of 1000 fooling images (adversarial samples) by attacking some images from the training set of ImageNet. Afterwards, we then fine-tune the model (i.e., VGG-19) including a set of adversarial examples (1000), in addition to a sample of 34,745 images from the ImageNet training set. Finally, we test the ASR on the test dataset to check if the adversarial training was effective. For VGG-19, we found that after re-training with this augmented dataset and testing with the set of roughly 3K fooling images, the  drops to approximately 5.33%.




§ RELATED WORK








Adversarial backdoor attacks.
Backdoor attacks[In the literature, one can also find the term trojan attack. These two terms can be used interchangeably.] on neural networks have been gaining attention in the adversarial learning community <cit.>.
While the first works focused on adding additional hardware components to plant the backdoor <cit.>, most of the follow-up works utilized changes in the training data to inject the backdoor <cit.> stealthily.
These attacks work by poisoning training samples to encode backdoor functionality during the training process <cit.>.
The backdoor is then activated by adding a trigger to the model input, which generally causes misclassification to a target class.
It was shown that this trigger can be hard to distinguish by the human eye <cit.>.

Fault-based backdoor attacks.
One of the challenges of the poisoning-based backdoor attack is that the adversary needs to be able to tamper with the training data, which might be impractical in many scenarios.
The answer to that are the fault-based backdoor attacks that modify the model parameters directly without changing the training data.

    
  * Inference phase attacks. These attacks change the model weight after the training.
    The first attack in this direction presented a generic methodology for backdooring CNNs via targeted weight perturbations <cit.>. 
    As perturbing weights has been previously used for misclassifying outputs <cit.>, this hardware attack method was utilized by Rakin et al. <cit.> to plant a backdoor at the bit-level.
    After that, several works adopted this method in various contexts <cit.>.
    Generally, weight-flipping attacks assume the attacker can use Rowhammer <cit.> – a bug in random access memories that allows bit-flipping of target bits by rapidly rewriting the surrounding area of the memory.
    
  * Training phase attacks. There are currently two directions in this sub-category, each of them having a single representative as of now.
    The first direction is similar to the aforementioned attacks in the sense that the attacker utilizes Rowhammer to flip the bits – but this time during the model training.
    DeepVenom is an instantiation of such an attack <cit.>.
    However, compared to DeepBaR, DeepVenom requires a visible large trigger image embedded in the original image and therefore is not imperceptible to human eyes.
    Our approach falls within the second direction – tampering with the processor instructions during the training process.
    The closest work to ours is an attack called FooBaR <cit.> which inserts a backdoor during the training phase of the model by faulting activation functions.
    However, the generation of triggers for FooBaR requires constraint solving, which is generally an NP-complete problem.
    The solving complexity grows exponentially with the number of parameters, and therefore, does not scale to large networks that are used for image classification tasks such as CIFAR10 or ImageNet.
    Our approach, on the other hand, generates the fooling images with gradient descent and thus can be used on arbitrarily large networks.
    Another disadvantage of FooBar is that, unlike in DeepBaR, the fooling images are visibly different from the original ones.


To put our work into the context of backdoor attacks, we can use a classification chart shown in Figure <ref>, inspired by <cit.> but with two additional categories: attack method, and attack phase.
The attack method differentiates between the poisoning attacks, detailed at the beginning of this section, and fault-based attacks, where  belongs.
Because of that category, we had to add the attack phase as well – while the poisoning attacks are always performed during the training, the bit-flip-based fault-based backdoors can also be injected during the inference.
In Figure <ref>, the categories where our attack belongs are highlighted in red color.






Defenses against backdoor attacks.
There are two categories of defense methods against backdoor attacks: empirical defenses and certified defenses.
Empirical defenses <cit.> are based on characteristics of existing attacks and provide good effectiveness against those.
However, as there is no theoretical guarantee with this type, it is possible to adapt attacks in a way to bypass them <cit.>.
Certified defenses, on the other hand, provide provable security guarantees under some assumptions <cit.>.
The performance of these is generally weaker compared to empirical defenses as it is hard to fulfill the given assumptions.

It is important to note that the existing defenses against backdoors protect against poisoning-based attacks, and therefore do not effectively protect against fault-based attacks <cit.>.

Defenses against fault-based attacks.
As the majority of fault-based attacks are weight-oriented, most of the defenses focus on protecting against bit flips in memory.
Techniques in this direction include checksums <cit.>, hashing <cit.>, encoding <cit.>, usage of binarized neural networks <cit.>, or in-DRAM swapping <cit.>.
A method called NeuroPots <cit.> utilizes a proactive honeypot approach to “lure” the attacker to flip certain bits in the network, making the detection and recovery more efficient.

To develop an appropriate countermeasure against ReLU-skip attacks, one would have to turn to techniques aimed at cryptographic implementations, as control-flow integrity protection is well-studied in this domain <cit.>.
One can use software methods such as instruction duplication/triplication to provide some level of protection <cit.> or replace a vulnerable instruction with a sequence of fault-tolerant ones <cit.>.
In hardware, redundant circuits can be used to some extent – however, it was shown that with precise equipment, one can simultaneously inject identical faults into these circuits <cit.>. 

Another class of countermeasures utilizes a separate device, usually a sensor <cit.>, that can detect local disturbances of a potentially harmful character.
For a more complete overview of techniques used for protecting cryptographic implementations, an interested reader is suggested to look at <cit.>.




§ CONCLUSIONS



In this work, we introduce a novel fault-based backdoor attack, , that allows attackers to trigger targeted misclassification for several image-based neural network architectures (such as ResNet-50, VGG-19, and DenseNet-121). Our technique relies on simple fault injection techniques that are applied during training (i.e., fine-tuning) stages. Overall, our approach achieves greater attack success rates than state-of-the-art targeted adversarial techniques without compromising the performance of the benign pre-trained model. Remarkably, our technique does not require full training of complex surrogate models to create further adversarial examples. Moreover, our approach requires 50 times fewer queries to the model than other backdoor attacks when creating adversarial examples. Additionally, we have presented a countermeasure to protect deployed systems against our proposed fault-based attacks.


IEEEtran




