Paper Title: Quality Control for Radiology Report Generation Models via Auxiliary Auditing Components.









Error Detection for Report Generation Models




H. Warr et al.

Department of Engineering Science, University of Oxford, Oxford, UK 

{first_name.last_name}@eng.ox.ac.uk
Department of Oncology, University of Oxford, UK
Department of Medical Physics and Clinical Engineering, Oxford University Hospitals NHS FT, Oxford, UK
Department of Computing, Imperial College London, London, UK
School of Computer Science, University of Birmingham, Birmingham, UK








Quality Control for Radiology Report Generation Models via Auxiliary Auditing Components
    Hermione Warr1, 
Yasin Ibrahim1, 
Daniel R. McGowan2,3, 
 
Konstantinos Kamnitsas1,4,5
    August 5, 2024
==========================================================================================



§ ABSTRACT 
Automation of medical image interpretation could alleviate bottlenecks in diagnostic workflows, and has become of particular interest in recent years due to advancements in natural language processing. 
Great strides have been made towards automated radiology report generation via AI, yet ensuring clinical accuracy in generated reports is a significant challenge, hindering deployment of such methods in clinical practice. 
In this work we propose a quality control framework for assessing the reliability of AI-generated radiology reports with respect to semantics of diagnostic importance using modular auxiliary auditing components (ACs).
Evaluating our pipeline on the MIMIC-CXR dataset, our findings show that incorporating ACs in the form of disease-classifiers can enable auditing that identifies more reliable reports, resulting in higher F1 scores compared to unfiltered generated reports. 
Additionally, leveraging the confidence of the AC labels further improves the audit's effectiveness.
Code will be made available at: <https://github.com/hermionewarr/GenX_Report_Audit>








§ INTRODUCTION

 
Radiologists are facing mounting challenges managing the growing volume of medical imaging data under resource constraints, necessitating more efficient and innovative solutions for image analysis and interpretation. Automated (or partly automated) radiology report generation is one such solution.
Since the introduction of transformers <cit.>, which significantly enhanced language generation, numerous studies have investigated radiology report generation from medical images, particularly chest X-rays (CXR) <cit.>.
However, deployment of language models to medical applications faces a variety of challenges, with a key area being the need for factual accuracy. 
Various methods have been developed to address this - particularly useful have been the introduction of clinical metrics to supplement the standard natural language processing (NLP) metrics used to evaluate report generation quality <cit.>.
Previous efforts to improve the quality of the generated reports have incorporated extra information during training, such as bounding boxes for improved localisation, or have aimed to refine diagnostic accuracy by including loss functions that reward clinical accuracy <cit.>.
Additionally, more complex architectures, such as memory-augmented attention, have been developed to facilitate better image understanding <cit.>. 

Besides efforts to improve report generation accuracy, another line of work is error detection in language modeling. Most work in this area has targeted the natural language domain. The two main types of uncertainty estimation in large language models (LLMs) are statistical uncertainty (SU) and in-dialogue uncertainty (IDU) <cit.>. The latter relies on large pretrained models to admit when they don't know an answer <cit.>.
SU relates certainty to the model's output entropy, though in language modelling, it must address the coherence of entire sentences, as well as on a per token basis <cit.>. These methods often depend on extensive labelled datasets in the natural language domain, which do not adequately capture the specialised terminology and context of clinical settings <cit.>. 

Few works have investigated the possibility of error detection in reports written by radiologists <cit.>. 
To our knowledge, no previous work has investigated error detection for automated report generation, which is the focus of our study.


We introduce an auditing framework for identifying potential errors in AI-based radiology report generation, using modular auxiliary auditing components (AC) for quality control. Our study focuses on chest-Xray (CXR) imaging, defining ACs as image-based disease classifiers to extract semantics relevant to diagnosis. We develop a report generation model, GenX, which performs competitively with previous works and serves as basis for testing our auditing framework. Reports are audited using ACs that provide disease classification and confidence levels. Reports with consistent diagnoses across the pipeline meet our criteria, while inconsistencies flag potential errors. Additionally, by leveraging the ACs' classification confidence, we enforce stricter quality control, requiring consistency with the highest confidence predictions. Our evaluation on MIMIC-CXR shows a significantly higher F1 score for reports meeting the audit's criteria. Results demonstrate that consistency between report and auxiliary components is a promising auditing mechanism for automated report generation.



§ METHODOLOGY







 §.§ Report Generation Model - GenX

The proposed framework for auditing radiology report generation is shown in Fig. <ref>. The backbone of the framework is the report generation model. The generalised vision language modeling problem of generating a textual sequence T given an input image I can be framed as estimating the conditional distribution p(T|I), as a product of the conditional probabilities:

    p(T|I) = ∏_l=1^L p(T_l | T_<l, I; θ),

where T is modelled as a sequence of word tokens {T_1,...,T_L}, with T_<l being the set of tokens preceding T_l, where L is the number of tokens in the sequence, and θ the model parameters <cit.>.

An overview of our report generation model, GenX, is shown in Fig. <ref>. The model is trained to maximise the probability of generating the target sequence from the image, p(T|I) from eq <ref>.
Image features I_feat=f_I(I) ∈ℝ^D_I, with dimensionality D_I, are extracted by image encoder f_I.
A linear layer, f_I,emb, projects I_feat to the space of token embeddings that will be input to the language generator, giving I_emb={I_emb,1 ... I_emb,M}=f_I,emb(I_feat) ∈ℝ^D_T × M, a set of M image token embeddings with dimensionality D_T, similarly to LLaVA <cit.>. 
A tokeniser maps text to a set of L token vectors {T_l}_1:L∈ℝ^L × V, where V is the vocabulary size. Each T_l is embedded by a linear layer to T_l,emb=f_T,emb(T_l) ∈ℝ^D_T.
The concatenated embeddings of image and text tokens {I_emb,1 ... I_emb,M, T_emb,1 ... T_emb,L} are passed to a Transformer decoder (Fig. <ref>b) which predicts a probability distribution over V vocabulary tokens, expressing which is more probable in the sequence. The decoder has N_layers with Attention blocks that each has N_heads parallel heads <cit.>.
The model is trained using the common paradigm of diagonal attention masking <cit.>.
A report T_G is generated with auto-regressive inference, starting only with image token embeddings I_emb as input, then predicting the first and each following word token as the one with highest predicted probability.








 §.§ Auditing Generated Reports via Auxiliary Classifiers



While some types of mistakes in AI-generated radiology reports, such as grammatical errors, are merely inconvenient, semantic mistakes related to disease diagnosis can be detrimentalor patient care. 


Consider a semantic concept of interest, denoted c. We focus on the case where c represents a specific disease, with the class label C∈{1,0} indicating the presence or absence of that disease. 
To audit whether a generated report T_G is semantically correct about c, 
we need to extract the value C_T described in text T_G via a mapping function g_T: T_G → c. This can be any text-based classifier, or prompt-based LLM that can answer questions in the form "Does this text suggest evidence or not for disease c?", which have become widely available. Herein we use a pretrained CheXbert text-classifier <cit.> as g_T, obtaining C_T = g_T(T_G) (Fig <ref>b).

We then introduce to the framework an independent model g_I: I →c, which infers value C_I for concept of interest c from image I (Fig. <ref>c). We term such models auxiliary auditing components (AC). In the examined case when c is existence of specific disease, g_I is a classifier predicting C_I=1 if image I shows the disease or C_I=0 otherwise, with model confidence p_AC(c=C_I |I).
If multiple concepts c are of interest (e.g. different diseases), an independent AC model can be used per c, for ease of modular development, or a multi-label AC model.

Auditing is performed by comparing values C_T and C_I about the concept c of interest (e.g. disease) extracted from the report or the image respectively (Fig <ref>d). To consider the contents of the report reliable with respect to c, agreement between the report and AC is required, i.e. C_T = C_I. We further extend this by taking into consideration the confidence p_AC of the AC. We only consider the auditing successful if it satisfies the additional requirement that the value C_T=C_I has been inferred by the AC with confidence p_AC over a pre-determined threshold t. Formally, for a successfully completed auditing we require: 

    (C_I = C_T)  (p_AC(c = C_I | I) ≥ t),

where  is the logical and operator. 

The set of generated reports that pass the audit for a particular concept c is denoted by T_G,pass^(c)={T_G |  Eq. <ref> is  True}. The remainder, T_G,err^(c)={T_G |  Eq. <ref> is  False}, can be deferred to the human user for review as potentially inaccurate for that disease.

The framework is designed based on ACs that are modular, independent models. The design draws inspiration from N-Version programming <cit.> and the principle of redundancy for reliability in software engineering. Assuming independence of components (GenX, g_T, g_I) with label error rates e_C_T and e_C_I, the error rate of the auditing framework 
is e=e_C_T· e_C_I, which is lower than the individual error rates of C_T and C_I, as failure of auditing requires both components to fail simultaneously to incorrectly satisfy Eq. <ref>.
Modular ACs enable independent component training and validation prior to integration into the audit pipeline. 



§ EXPERIMENTS AND RESULTS



 §.§ Data



Experiments are performed using MIMIC-CXR <cit.>, containing chest X-rays (CXR), associated radiology reports, and disease labels for ∼228,000 studies of over 65,000 patients.
As common practice, we consider only the frontal-view images, and only the “findings” section of the reports, which describes evidence of pathologies and support devices. 
The labels 
include 14 classes (12 diseases, 1 support devices, 1 no findings) and
can be positive, uncertain, negative, or not mentioned. 
We report 5 out of 14 classes separately (Atelectasis, Cardiomegaly, Consolidation, Edema, Pleural Effusion) to facilitate comparison with literature <cit.>. We divide data into splits for training (226,261), validating hyper-parameters (1,864), and testing performance (3,595). 
To train and evaluate the report generation, we further exclude studies that do not include a “findings” section, resulting in 155,322 training, 1,231 validation and 2,607 testing samples. Finally, to help address class imbalance and noisy labels, following past works <cit.>,
we consider uncertain disease labels as positive and class no-mention as negative.




 §.§ Evaluating the Report Generation Model


Experimental settings: We here show that our report generation model, GenX, is of representative quality in comparison to previous work to serve as effective backbone for the study of our auditing framework. 
To build GenX we use ResNet-50 as the image encoder f_I, which we pre-train as a multi-label classifier of the 14 classes described above. Its penultimate layer of dimension D_I=512 is then projected to M=10 image token embeddings with D_T=512. These are concatenated with up to L=512 text embeddings, resulting from a GPT-2 tokeniser (V≈50,257) <cit.>, followed by a projection, D_T=512. The Transformer has N_layers=8, N_heads= 8, and dimension D_ff= 2048 of its feed-forward layer. GenX is then trained end-to-end (Fig. <ref>), learning parameters of the randomly initialised projections and Transformer, and fine-tuning the f_I encoder.



Results: 

Table <ref> reports test performance by our report generation model, along with previous works.

NLP metrics BLEU (BL) <cit.>, METEOR (M) <cit.> and  (R-L) <cit.> assess generated language quality. 
More important for this study is the assessment of diagnosis-related semantics in generated reports, using disease-classification metrics such as F1 score, by comparing agreement of class labels, C_T, extracted from generated and reference reports via CheXbert <cit.>.
Our report generator, GenX, achieves NLP metrics similar to  Ratchet <cit.>, which has the most comparable architecture. Importantly, GenX achieves a high F1 score in comparison to Ratchet, R2Gen <cit.>, even RGRG <cit.> that needs additional labelled bounding boxes for training. It approaches F1 score by ℳ^2 Trans <cit.> that trains using a more complex framework and reinforcement, and is surpassed by A_tTERG<cit.> which uses extra labels for training.
We conclude that GenX produces reports of sufficient quality to serve as effective backbone for the following study of our auditing method. Two examples of generated reports are shown in Table <ref>. The generated reports use realistic language, though they can occasionally miss a diagnosis. Thus a framework that can extract and audit report semantics is crucial.





 §.§ Evaluating the Auditing Framework for Generated Reports


Experimental settings: 
We analyse whether including auxiliary auditing components, ACs, enable us to quality control the generated reports and filter out those that are more likely to contain errors.
We evaluate on the 5 diseases most commonly reported in the literature (Sec. <ref>), from the CheXpert competition  <cit.>. For this, we first train 5 independent disease-specific AC image classifiers (AC_1), where each is a ResNet-18. 
We then apply the framework as described in Sec. <ref> to the test-split. For each image, the 5 AC_1 models predict 5 disease labels C_I^(c), one per disease c.
GenX generates a report for each test image and we extract 5 report-based class labels C_T^(c) via Chexbert, for each disease.
For a given scan, per disease c, we compare image and report based labels, C_I^(c) and C_T^(c), to check if they fulfil Eq. <ref>. We perform the check in two settings: First, without considering a confidence threshold for p_AC, satisfying only the first condition in Eq. <ref>; Second, with a confidence threshold t=0.8 where Eq. <ref> is fulfilled if p_AC≥0.8, to investigate whether stricter auditing criteria results in more accurate reports
For each disease, the images that fulfil the above criteria are considered to have successfully passed the audit. For these images, we calculate average F1 score by comparing their report-based labels C_T^(c) with the reference report labels, to determine whether the average score is higher than that of all unfiltered reports generated by GenX.
Finally, we repeat these experiments using a bigger ResNet-50 AC, trained as a multi-label image classifier of all 14 classes (AC_14). This is to evaluate whether it's beneficial to integrate a larger AC that learns from all 14 labels available per scan, or independent, disease-specific ACs. 



Results: Table <ref> shows the results. Class labels C_T extracted from reports generated by GenX achieve an average F1 score of 46.8 over the 5 diseases. Reports that agree with AC_1, fulfilling the auditing without the extra requirement for confidence p_AC≥t, achieve average F1 score 54.3. These benefits are realised over a significant percentage of the reports (72-80% across the diseases), while the rest would be deferred for user inspection and error correction in an actual workflow.
Reports that pass the stricter auditing and 
additionally fulfil the confidence requirement p_AC≥0.8 present even less errors, with average F1 score 58.4. This restricts the number of reports that pass the audit (30-65%), in return for higher reliability.
In comparison to the unfiltered reports of GenX, the reports that fulfil the audit show increased F1-scores for 4 out of 5 studied diseases. The exception is Consolidation, the class where both GenX and AC image classifiers have very low performance. Interestingly, for 3 out of 5 diseases, the reports that successfully pass the audit show F1-scores that are even higher than the auditing AC_1 classifiers.
These results show that measuring consistency between semantics of reports and auditing classifiers is a promising framework.

Finally, we find that a single, multi-label classifier, trained for all 14-classes as AC_14, achieves significantly lower F1 scores for image classification, which also leads to lower effectiveness of the auditing. Although AC_14 is a ResNet-50, larger than ResNet-18s used for the 5 AC_1 classifiers,

it is difficult to develop it at the level of disease-specific classifiers. This demonstrates the benefits of modularity, with independently developed AC for each concept of interest.





§ CONCLUSION

We introduced a framework to quality control AI-generated radiology reports with respect to semantics of interest via auxiliary auditing components. 
Our study focused on the task of report generation from chest-Xrays, detecting potential semantic errors in diagnosis-related report content, using auxiliary disease classifiers and assessing consistency between labels inferred from image and report.
Experiments show that reports fulfilling the auditing criteria exhibit fewer errors, with average F1 score that can even exceed the score of the auditing components. The framework is generic and future work could explore it for auditing semantic concepts beyond disease classification, such as for description of pathology location or volume, using regression models as ACs for such properties.

The limitation of the framework, as with any filtering pipeline for quality-control, is the inherent trade-off between reducing the number of reports that successfully pass the audit and can be used for down-stream workflow, in exchange for their higher reliability. Future work could improve performance in both aspects by integrating more potent models for inferring image-based and report-based labels, thereby ensuring greater consistency and reliability from their comparison.




§ ACKNOWLEDGEMENTS

Hermione Warr and Yasin Ibrahim are supported by the EPSRC Centre for Doctoral Training in Health Data Science (EP/S02428X/1). 








splncs04





 Paper Title: Small Object Few-shot Segmentation for Vision-based Industrial Inspection.





Enhancement of the molecular electron chirality by electronic excitation

    Masato Senami
    August 5, 2024
==========================================================================




§ ABSTRACT 
Vision-based industrial inspection (VII) aims to locate defects quickly and accurately. Supervised learning under a close-set setting and industrial anomaly detection, as two common paradigms in VII, face different problems in practical applications. The former is that various and sufficient defects are difficult to obtain, while the latter is that specific defects cannot be located. To solve these problems, in this paper, we focus on the few-shot semantic segmentation (FSS) method, which can locate unseen defects conditioned on a few annotations without retraining. Compared to common objects in natural images, the defects in VII are small. This brings two problems to current FSS methods: 172 distortion of target semantics and 173 many false positives for backgrounds. To alleviate these problems, we propose a small object few-shot segmentation (SOFS) model. The key idea for alleviating 172 is to avoid the resizing of the original image and correctly indicate the intensity of target semantics. SOFS achieves this idea via the non-resizing procedure and the prototype intensity downsampling of support annotations. To alleviate 173, we design an abnormal prior map in SOFS to guide the model to reduce false positives and propose a mixed normal Dice loss to preferentially prevent the model from predicting false positives. SOFS can achieve FSS and few-shot anomaly detection determined by support masks. Diverse experiments substantiate the superior performance of SOFS. Code is available at https://github.com/zhangzilongc/SOFShttps://github.com/zhangzilongc/SOFS.




§ INTRODUCTION



Vision-based industrial inspection (VII) automatically identifies defects in industrial processes and ensures product quality. Compared to object recognition in natural images, defect recognition in VII is fine-grained. As shown in Fig.<ref>, fine-grained defects appear as small objects, the area proportions of defects are mostly less than 0.3%, and the smallest is even 0.015%. 

The common paradigm in VII is supervised learning under a close-set setting <cit.>. This paradigm requires researchers to collect a large number of various defects in specific scenarios, and then design specific models for fitting the distribution of small defects in training. Since industrial processes are generally optimized to minimize the production of defective products <cit.>, collecting various and sufficient defects for training is often intractable, which makes it difficult for such methods to be widely applied. To tackle this problem, industrial anomaly detection (IAD) <cit.> has been widely adopted recently. IAD only requires normal samples to detect defects, while there are a large number of normal samples. Such a characteristic is conducive to the application of IAD in real scenarios. However, due to limitations of the problem formulation of IAD, it regards various defects as the same class, , abnormal class, which prevents it from locating specific defects. In real industrial processes, different defects have significantly different impacts on products, , scratches on the printed circuit board affect the appearance, while broken circuits affect the use. Ablation of aircraft engine blades affects engine efficiency, and cracks may cause engine fire <cit.>. Thus, locating specific defects has significant practical value. 



To break through the above limitations, this paper focuses on the few-shot semantic segmentation (FSS) method. FSS divides the input into the query and support sets following the episode paradigm <cit.>. It segments the query targets conditioned on the semantic clues from a few support annotations and can quickly adapt to unseen classes without retraining. These properties make FSS an ideal approach for VII. The achievement of previous FSS methods <cit.> is built upon the accurate extraction of semantic features of the support set, determined by the feature intensity in the downsampling of support annotations and the information in the resizing image. As shown in Fig. <ref>, applying these methods to segment small defects brings two problems: failure to segment and many false positives for backgrounds. The former is due to the distortion of target semantics, caused by the information loss during the resizing of the original image and the feature intensity bias in the downsampling of support annotations. The latter is caused by the missing defect, , the first and third rows in Fig. <ref>, and the train-set overfitting.



In this paper, to alleviate the above problems, we propose a small object few-shot segmentation (SOFS) model for VII. SOFS alleviates the distortion of small objects by 1) avoiding the resizing and 2) correctly indicating the intensity of target semantics. Specifically, 1) SOFS uses a non-resizing procedure to crop small objects in training and adopt the sliding window mechanism in the test. SOFS ensures that the pixel area of small objects encoded by the model is consistent with that in the original image. 2) We propose the prototype intensity downsampling of support annotations to correct the bias caused by the common interpolation. This downsampling is calculated by all pixels of corresponding regions of target semantics, providing a better estimate of target semantics.

To reduce false positives caused by the missing defect and the train-set overfitting, 1) We design an abnormal prior map in SOFS to guide the model to highlight the semantic while ignoring the normal background. This design also enables SOFS to have both FSS and few-shot anomaly detection (FAD) abilities. 2) We propose a mixed normal Dice loss to impose a large penalty when the model predicts false positives, preferentially preventing the model from predicting false positives.













Contributions: 1) We recognize FSS as an ideal solution for addressing the difficulty of obtaining diverse defects and the inability of anomaly detection to identify specific defects in VII. 2) We propose SOFS to alleviate the distortion of target semantics and many false positives. The key idea of the former is to avoid the resizing of the original image and correctly indicate the intensity of target semantics. The latter is to guide the model to focus more on backgrounds. SOFS can adapt to unseen classes without retraining, achieving FSS and FAD. To the best of our knowledge, SOFS is the first model to realize both FSS and FAD in VII. 3) FSS experiments on VISION V1 show that SOFS achieves state-of-the-art results, achieving 12.2% mean mIoU improvements compared to SegGPT. FAD experiments also demonstrate its competitive performance.



§ RELATED WORK





 §.§ Few-shot Semantic Segmentation


Few-shot semantic segmentation <cit.> predicts dense masks for novel classes with only a few annotations. According to the different fusion granularity of support features and query features, previous methods can be divided into image-level feature fusion <cit.> or pixel-level feature fusion <cit.>. The image-level feature fusion methods extract the semantic clue of support sets by a masked average pooling of a global image and then fuse this prototype with the query features. Since a pooling prototype feature cannot cover all regions of an object, the pixel-level feature fusion methods are proposed to mine the correspondence between the query pixel-level features and the support semantic-related pixel-level features, where the residual connection in the cross attention plays the role of fusing query and support features.



Previous methods mostly focus on the segmentation of PASCAL-5^i <cit.> and COCO-20^i <cit.>, where the segmented objects are mostly salient. In comparison, the segmented objects in VII are generally small. In VISION V1 <cit.>, 84% of defect types have area proportions less than 0.3%, and 38% of defect types have pixel areas less than 900. These characteristics either cause previous methods to predict many false positives or fail to segment.





 §.§ Small Object Recognition


The term “small object” refers to objects that occupy a small fraction of the input image <cit.>, , in the widely used MS COCO dataset <cit.>, it defines objects whose bounding box is 32× 32 pixels or less in a typical 480 × 640 image, while other datasets define the objects that occupy 10% of the image. The main challenges for small object recognition include information loss, low tolerance for bounding box perturbation, etc <cit.>. Information loss refers to the fact that the feature information of small objects is almost wiped out during the downsampling of the feature extractor, it has the greatest impact on performance. To alleviate this issue, there are mainly three kinds of methods. The first is to maintain the high-resolution feature map while retaining the fast process <cit.>. The second is a super-resolution-based method <cit.>, which restores the distorted structures of small objects. The third is to process small objects by multi-scale learning and hierarchical feature fusion <cit.>.

Previous methods mainly focus on small object recognition in a close-set setting. In this paper, we study this in an open-set setting, the target is to learn the meta-class-agnostic feature.







 §.§ Few-shot Anomaly Detection (Abnormal Segmentation)


Recently, few-shot anomaly detection (FAD) <cit.> has been proposed to simultaneously solve the problem of difficulty in collecting various and sufficient defects in VII and the problem of requiring retraining for different objects. FAD can quickly adapt to unknown classes by a few normal samples, , support images. FAD methods detect anomalies based on the similarity of pixel-level features between the query image and the support images. Furthermore, recent works <cit.> add textual descriptions to supplement the missing details of the image modality.

Anomaly detection regards various defects as the same class, , abnormal class, which prevents it from identifying specific defects. In VII, different defects have significantly different impacts on products, which makes anomaly detection unsuitable. In this paper, our proposed method can simultaneously achieve FSS and FAD and is applicable in more industrial scenarios.



§ METHODS





 §.§ Problem Formulation


Few-shot semantic segmentation trains model to capture the meta-knowledge of the training set, adapting to novel objects with only a few annotated support images. In definition, the model is trained on the training set D_train and tested on the test set D_test. Suppose the category sets in D_train and D_test are C_train and C_test respectively. In the few-shot setting, C_train∩ C_test=∅. Episodes are applied to D_train and D_test, each episode consists of a query set Q={(I^q, M^q)} and a support set S={(I^s_i, M^s_i)}^K_i=1 with the same class, where I^q, I^s_i∈ℝ^H̅×W̅× 3 represent the RGB images and M^q, M^s_i∈{0, 1 }^H̅×W̅ denote their binary masks, K denotes the number of support images. In training, both M^q and M^s_i are used, while only M^s_i are accessible in testing. During testing, the model requires no optimization for the novel classes. In FAD, M^s_i∈{0 }^H̅×W̅.





 §.§ Small Object Few-shot Segmentation



  §.§.§ Overview


The proposed small object few-shot segmentation (SOFS) model consists of three major steps: 1) small object enhancement designs, 2) feature augmenter, 3) feature fusion and meta prediction, as illustrated in Fig. <ref>, where the small object enhancement designs alleviate the distortion of small objects. In training, SOFS crops the semantic area on the support images and query images (Section <ref>). The cropped query/support images are then encoded by a feature augmenter to generate the semantic-related features (Section <ref>). Afterward, the encoded query and support features are fused via a non-learnable feature fusion. Finally, the fused feature interacts with the prototype features to produce the query mask (Section <ref>). In testing, SOFS uses a sliding window mechanism to produce multiple query images. To reduce false positives caused by the train-set overfitting, we elaborate on a mixed normal Dice loss (Section <ref>).





  §.§.§ Small Object Enhancement Designs





Non-resizing Procedure. The core idea is to ensure that the pixel area of small objects encoded by the model is consistent with that in the original image. As shown in Fig. <ref>, the non-resizing procedure randomly crops the small object on the original image in training and uses the sliding window mechanism to process all regions of the query image in the test. The reason why the non-resizing procedure can be used is that the segmentation of small defects in VII has a strong correlation with the surrounding areas, but a weak correlation with distant areas, which eliminates the need for the model to establish long-range dependencies. In addition to the above benefits, the non-resizing procedure can produce normal samples (the non-defective areas are all normal areas). We show in Section <ref> how these normal samples alleviate model overfitting. 

In the following part, we refer to query set and support set after the non-resizing procedure as Q={(I^q, M^q)} and S={(I^s_(1), M^s_(1))}, where we show the case K=1 without loss of generality.



Prototype Intensity Downsampling. To extract the prototype feature on the support image, we need to downsample the support mask M^s to ensure that it is consistent with the size of the support feature map. As shown in the patch-wise original mask of Fig. <ref>, every patch represents a region that is highly correlated with corresponding features on the feature map, since the segmented object is small, some corresponding features of the feature map may be weak, occupying only a small part of the original segmentation area. However, if we use the common bilinear/bicubic interpolation to downsample the original mask to indicate the feature intensity, since the bilinear/bicubic interpolation only uses 4/16 points to get the result, the result may be overestimated or underestimated, as shown in Fig. <ref>, which leads to the distortion of target semantics. To alleviate this issue, we propose a prototype intensity downsampling to replace the common bilinear/bicubic interpolation in the downsampling of M^s. Specifically, we employ a l× l convolution layer to process M^s: 


    M̂^s=Conv(M^s)/l× l ,


where M̂^s∈ℝ^H× W denotes the downsampling of the support mask, l denotes the multiple of downsampling of feature map, the parameters in the convolution kernel are all 1, the stride of convolution is l. M̂^s is calculated by all pixels of corresponding regions of features. It provides a better estimate of the intensity of prototype features, avoiding the mismatch of semantic clues.





  §.§.§ Feature Augmenter



Firstly, we encode I^s and I^q separately, resulting in feature maps F^s, F^q∈ℝ^H× W× C. Subsequently, we use a prior generation module to generate the prototype feature p∈ℝ^C, a semantic prior map M^q_s∈ℝ^H× W and an abnormal prior map M^q_a∈ℝ^H× W. p is extracted by a MaskAvgPool(F^s,M̂^s), where MaskAvgPool(·) denotes the masked average pooling (detailed descriptions are provided in Appendix). M^q_s and M^q_a are formulated as follows:










    M^q_s=Ψ(Φ(F^q) Φ(F^s)^T⊙φ(M̂^s)), 
    
    M^q_a=1-Ψ(Φ(F^q) Φ(F^s)^T⊙ (1-φ(M̂^s))),


where Φ(·): ℝ^H× W× C↦ℝ^HW× C refers to the reshape function and L_2 normalized on feature dimension, φ(·): ℝ^H× W↦ℝ^1× HW↦ℝ^HW× HW refers to the reshape and repeat along the row, Ψ(·): ℝ^HW× HW↦ℝ^HW× 1↦ℝ^H× W refers to taking the maximum value along the column and the reshape. M^q_s denotes the maximum similarity between each pixel-level query feature and support semantic features. The larger the value on M^q_s, the more relevant it is to the semantics in the support set. 1-M^q_a denotes the maximum similarity between each pixel-level query feature and support normal features. The larger the value on M^q_a, the smaller the similarity with normal features in the support set, and the more abnormal the corresponding region is.



The motivation for adding M^q_a is that we hope that SOFS can reduce false positives for backgrounds in VII. Specifically, in VII, there is a unique type of defect: the missing defect, where the target area misses some components, , the defect shown in Fig. <ref> is a missing defect, we can only observe the image background. If we only use semantic matching for the missing defect, it may highlight many false backgrounds. M^q_a matches every pixel-level query feature with the normal support features, if there is a missing defect, it can be highlighted and the normal background can not be. In addition, M^q_a enables SOFS to have FAD ability, we can input the normal support image.

After obtaining the above features, we concat F^s with p, concat F^q with p, M^q_a, and M^q_s, then use a 1× 1 convolution to produce F̅^s, F̅^q∈ℝ^H× W× C_1. Subsequently, we use the self-attention <cit.> to further improve F̅^s, F̅^q.



  §.§.§ Feature Fusion and Meta Prediction



F̅^s, F̅^q are subsequently fused by a non-learnable feature fusion. The matching mechanism follows <cit.> to replace the dot produce with the cosine similarities, which is formulated as follows:


    F̅^qs=softmax^'(Φ(F̅^q) Φ(F̅^s)^T/τ)Φ^'(F̅^s⊙ϑ(M̂^s))+Φ^'(F̅^q),


where Φ^'(·): ℝ^H× W× C↦ℝ^HW× C refers to the reshape function and non L_2 normalized, τ controls the distribution shape, softmax^'(·) refers to the normalization along the row, , reverse softmax <cit.>, ϑ(·): ℝ^H× W↦ℝ^H× W× C refers to first expanding the new dimension and then replicating along the expanded dimension. Eq. (<ref>) can be regarded as a type of cross-attention, where the learnable parameters are discarded. We think that the recognition of small objects does not need lots of parameters, more parameters may cause the risk of overfitting the category-specific information. After the feature fusion, we extract the prototype feature p̅∈ℝ^C_1 by a MaskAvgPool(F̅^s, M̂^s). Then the prediction result of SOFS model is formulated as follows:


    M̂^q_pred=𝕀(sum(M^s)>0) Sigmoid(ϕ(F̅^qsp̅))+(1-𝕀(sum(M^s)>0))M^q_a,


where 𝕀(·) refers to a indicator function, Sigmoid(·) is a sigmoid function, ϕ(·): ℝ^HW× 1↦ℝ^H× W refers to the reshape function. Compared with the learnable prediction classifier (a multi-layer perceptron) used in <cit.>, our meta prediction can be learned dynamically, it is encouraged to learn a class-agnostic meta feature. M̂^q_pred is a combination of a semantics prediction and an abnormal prior map, if the support images are all normal samples, SOFS produces a result of few-shot abnormal segmentation. Otherwise, SOFS produces a result of few-shot semantic segmentation.



  §.§.§ Loss Function



In our experiments, we find that training SOFS by Dice loss <cit.> and the cross entropy loss suffers from a severe overfitting problem, producing many false positives. To alleviate this issue, we propose a mixed normal Dice loss as follows:


    DiceLoss_normal(X, Y)= t(1-2 |X∩Y|/|X| + |Y |) + β(1-t)( 1- 1/η|X | + 1),


where t=𝕀(sum(Y)>0), X, Y refers to the prediction and the ground truth, η is a large hyperparameter, , 1e^4, β is a hyperparameter to control the weight, we set β=1 in the experiment. The normal sample term (the second term of Eq. (<ref>)) can be regarded as a regularization term of the original Dice loss. At the beginning of training, the model almost outputs 0 for all inputs. As the training evolves, once the model has a false prediction for the normal sample, the backpropagation of this term produces a larger gradient compared to defective samples, which preferentially prevents the model from predicting false positives. This loss facilitates that the model does not produce false positives while predicting the defects as much as possible. In our implementation, the normal query samples are from the non-defective areas in the non-resizing procedure, the probability of sampling normal query samples is α.

Our final loss function is a linear combination of the cross entropy loss and the mixed normal Dice loss, where the weight of the cross entropy loss is γ.



§ EXPERIMENTS





 §.§ Implementation Details


Datasets. FSS: We conduct experiments on VISION V1 <cit.> and Defect Spectrum <cit.>. VISION V1 <cit.> released in CVPR 2023 Vision-based InduStrial InspectiON workshop [https://vision-based-industrial-inspection.github.io/cvpr-2023/https://vision-based-industrial-inspection.github.io/cvpr-2023/], is one of the largest and most diverse manufacturing datasets in terms of both image number and available annotations. VISION V1 consists of 14 objects, spanning 44 defect types. 84% of defect types have area proportions less than 0.3%, and 38% of defect types have pixel areas less than 900.  Defect Spectrum <cit.> relabeled the data from MVTec AD <cit.>, DAGM2007 <cit.>, part of VISION V1 <cit.> and Cotton-Fabric <cit.>, including 97 defect types. Defect Spectrum is the dataset with the largest number of defect types so far. 63% of defect types have area proportions less than 2%. For the above datasets, cross-validation is conducted by dividing all objects into 3 folds, we ensure that the distributions of area proportions of defects in each fold are roughly the same. In the experiments, two folds serve as training data, while the remaining one is used for testing. To ensure performance stability and fairness for comparison, the query/support pairs sampling by all methods are the same. FAD:We use two benchmark FAD datasets, , MVTec AD <cit.> and VisA <cit.>, including 27 objects in total. More details of datasets and experiments are shown in Appendix.

Metrics. FSS: We adopt the mean intersection over union (mIoU) as the evaluation metric. We denote mIoU=1/CΣ^C_i=1IoU_i, where C is the number of defect classes in each fold, and IoU_i indicates intersection-over-union for class i. FAD: For classification, we report Area Under the Receiver Operating Characteristic (AUROC). For segmentation, we report pixel-wise AUROC and pixel-wise F_1-max.

Training and Test Details. FSS: We use ResNet-50 <cit.> and DINO v2 Base <cit.> as the encoder to verify the effectiveness of the proposed method and reimplementation methods on different backbones. The training input sizes of ResNet-50 and DINO v2 Base are set to 512× 512 and 518× 518 respectively. l in the prototype intensity downsampling are set to 8 and 14 for ResNet-50 and DINO v2 Base respectively. The probability α is set to 0.3, η is set to 1e^5, γ is set to 0.01, τ is set to 0.1. In the non-resizing procedure, the stride of sliding window and the input size in test are the same as the input size in training. All reimplementation methods and SOFS use the same training augmentations. FAD: We use DINO v2 Base <cit.> as the encoder. The input size is set to 518× 518. We only test SOFS. More details of SOFS and reimplementation methods are shown in Appendix.







 §.§ Comparison with State-of-the-Art Methods


FSS: In Table <ref>, we report the comparison of SOFS with other state-of-the-art FSS approaches on VISION V1. Compared with other methods, SOFS can gain passable improvement on ResNet-50. However, due to the insufficient discriminability of pre-trained features for small defects, the performance is extremely low. As we choose DINO v2 Base <cit.> as the encoder, the performance has a significant improvement and outperforms the others by a large margin. Despite using a model with fewer parameters, SOFS significantly outperforms SegGPT, achieving 11.4% (1-shot) and 12.2% (5-shot) of mean mIoU improvements. Notably, when applying the non-resizing procedure and the mixed normal Dice loss to comparison methods, the performance improves consistently, this illustrates the generality of the proposed method. In addition, we show qualitative results of SOFS in Fig. <ref>, <ref>, <ref>. Although there are different defect types in one image, SOFS can segment the specific small defects. Table <ref> shows the FSS performance on Defect Spectrum. Although this dataset contains larger defects than VISION V1, SOFS still achieves the best in most cases. The limitation of SOFS is that if the support images increase (K increases), the performance improvement for large defects is relatively weak (VISION V1 F2, DS-Spectrum F1 and F2). We think this is due to the underfitting to large defects. FAD: Table <ref> also shows the comparison of SOFS with other state-of-the-art few-shot anomaly detection approaches. Although SOFS only receives the image information, it can achieve a competitive performance with other multi-modal models. This shows that SOFS has strong capabilities for both FSS and FAD. 

In addition, we show the performance of FSS under domain shifts. We only test the model trained on VISION V1 <cit.> for Defect Spectrum <cit.> and AeBAD <cit.>, we verify the performance under different domain shifts. To avoid conflicts between test classes and training classes, we exclude VISION V1 from our test. Test defect types are 81. The results are shown in Table <ref>. Although these datasets contain larger defects than VISION V1 and different domain shifts, SOFS still achieves the best performance. More quantitative and qualitative results are shown in Appendix.







 §.§ Ablation Study


We report the ablation studies in this section to investigate the effectiveness of each component. All ablation experiments are conducted under VISION V1 1-shot setting with DINO v2 Base backbone.

Small object enhancement designs ablation. Table <ref> shows results regarding the effectiveness of small object enhancement designs. The baseline (line 1) uses the bilinear interpolation for M̂^s and the standard resizing procedure, its performance is extremely low. Then we add the prototype feature intensity map (PFIM), resulting in improved results. Since many defects account for small area proportions in F0 and F1, the performances of F0 and F1 are still low. Due to “Cylinder” with large defects in F2, the improvement for F2 is significant. Subsequently, we only add the non-resizing procedure (NR). Compared with line 2, this brings a greater improvement. However, the improvement in F1 is still not enough. This is mainly because there are many defects with small pixel areas in F1. NR cannot deal with this kind of problem. Finally, we use PFIM and NR to simultaneously improve the performance for defects with small pixel areas and small area proportions, resulting in a significant improvement for F1.

Other components ablation. Table <ref> shows results regarding the effectiveness of other components in SOFS. The baseline (line 1) is built upon the matching mechanism in <cit.> (a type of cross attention), the learnable prediction classifier (a multi-layer perceptron) used in <cit.> and trained by Dice loss <cit.>. We first replace Dice loss with the proposed mixed normal Dice loss (MNDL), resulting in a large improvement for F0 and F1. However, since the training on F2 does not include large defects (“Cylinder” with large defects is in F2), the model with too many parameters causes overfitting to small defects. Then we replace the matching mechanism in <cit.> with a non-learnable feature fusion, the performances for F0, F1, and F2 improve. Subsequently, we add an abnormal prior map (APM). As shown in Fig. <ref>, APM can reduce the false positive regions of missing defects, resulting in further improvements. Finally, we replace the learnable prediction classifier with a meta prediction, bringing 1.5% and 1.1% mIoU improvement for F0 and F1. But this also brings a slight degradation for F2. We find that this is due to insufficient prediction for large defects.





η and α in the mixed normal Dice loss. Fig. <ref> shows the results for different η and α. η: If η is small (0-10^-2), it does not penalize false positives enough, resulting in a relatively weak performance, especially for Fold 2. If η is set to 10^-1 or 10^0, the performance of Fold 0 and Fold 1 improve significantly. However, the performance of Fold 2 is 0. We found that after the training started, the model quickly fell into a local optimum with an output of all 0. This is mainly because the training for Fold 2 does not contain the large defects (Cylinder). In other words, if the training samples are all small defects, then some η will cause the training to fall into a useless local optimal solution. If η is set to a large number, this phenomenon will disappear. α: Different α have a significant impact on the results. If α is small, the number of normal query samples per iteration is small (possibly none), resulting in no penalty for false positives. If α is large, it causes the underfitting for defective samples.

r0.5
    

    
    < g r a p h i c s >

    

    The results as the test input size evolves in the sliding window mechanism.
    
    



Input size in test. Fig. <ref> shows the results as the test input size evolves in the sliding window mechanism (the stride is the same as the input size). We can find that different test input sizes have a significant impact on the segmentation of small objects, especially for the defects with small area proportions (Fold 0). The smaller the input size, the higher the latency of the model. We should choose the test input size based on actual requirements.



§ CONCLUSION



In this paper, we propose SOFS to tackle the problems existing in supervised learning under a close-set setting and industrial anomaly detection in VII. SOFS significantly improves the small object segmentation by avoiding resizing and a prototype feature intensity downsampling. Further, we add an abnormal prior map in SOFS to guide the model to reduce false positives for backgrounds, while enabling SOFS to realize both FSS and FAD. In addition, we propose a mixed normal Dice loss, non-learnable feature fusion, and meta prediction to alleviate the overfitting problem. The experiments for different defects under an open-set setting demonstrate the superior performance of SOFS. We hope that SOFS can provide some insights for the community of VII.

Limitations and Future Works 1) As shown in Fig. <ref>, the differences among defects in the same object are slight, which leads to some false positives. We think that test-time training/adaptation <cit.> may be a promising method to tackle this problem. 2) As shown in Fig. <ref> (7th row) and Fig. <ref> (3rd row), SOFS produces false negatives for defects with extreme aspect ratio. This may originate from the receptive field of the encoder. 3) If SOFS is used to process high-resolution images, due to the sliding window mechanism in the test, the latency of SOFS is high. We will explore some designs to reduce latency and segment defects with extreme aspect ratios in the future. 4) SOFS is allowed to adopt a mixture of normal support images and defective support images for training and inference, which is a more common scenario in reality. We will explore this direction in the future.





abbrvnat










§ APPENDIX




 §.§ Supplemental Details of SOFS


Masked average pooling. The masked average pooling is formulated as follows:


    MaskAvgPool(F,M)= sum^'(F⊙ϑ(M))/sum(M) ,


where F∈ℝ^H× W× C, M∈ℝ^H× W, ϑ(·): ℝ^H× W↦ℝ^H× W× C refers to first expanding the new dimension and then replicating along the expanded dimension, sum^'(·): ℝ^H× W× C↦ℝ^C refers to the addition of spatial features, sum(·): ℝ^H× W↦ℝ refers to the addition along spatial dimensions.  

Extension to K-shot setting. In extension to K-shot (K > 1) setting, K support images with their annoted masks S={(I^s_i, M^s_i)}^K_i=1 and the query set Q={(I^q, M^q)} are given. M̂^s_i, p_i, M^q_i_s and M^q_i_a are firstly calculated, where p_i is extracted by a MaskAvgPool(F_i^s,M̂^s), M^q_i_s denote a semantic prior map calculated by F_i^s and F^q. Then we stack all M^q_i_s and M^q_i_a along the number dimension of support set respectively. And we calculate the mean of maps concatenated toghter to obtain M^q_s and M^q_a. In the feature fusion, we calculate the cosine similarity between the query feature and all support features. We concat all F_i^s along the spatial dimension. The remaining steps are the same as 1-shot.

K-shot implementation. Due to the non-resizing procedure, we can sample multiple support images under a 1-shot scenario. These support images are from the random crops containing defects. In 1-shot experiments, we sample 4 random crops in one support image. In 5-shot experiments, we sample 1 random crop in one support image. We use this implementation for all methods.

Few-shot Anomaly Detection. We use the maximum value on M̂^q_pred to indicate the abnormal degree of the image.



 §.§ Supplemental Implementation Details


SOFS. SOFS is built upon the Pytorch <cit.> framework. All models are trained and tested on 8 NVIDIA GeForce RTX 2080 Ti GPUs. All reimplementation methods and SOFS use the same training augmentations, including rotation, blur, and flip. In the non-resizing procedure, we use the random crop, where the crop image includes the defect. We only use the non-resizing procedure if the area proportion of defect in the support set is less than 1%, otherwise, we use the common resizing. In addition, following the previous works, SOFS adopts features from different layers of backbones to further improve the useful information. 

SOFS is trained in an episode fashion for 50 epochs for VISION V1, the batch size is 4 for every GPU. During training, AdamW optimizer is adopted, and the learning rate is set to 1e^-5, the weight decay is 0.01, and the “poly” strategy is used to adjust the learning rate. During testing, predictions are resized back to the original sizes of the input images, keeping the groundtruth labels intact.

In the few-shot anomaly detection, SOFS adopts the common resizing procedure.

Reimplementation of Comparison Methods We compare SOFS with the recent methods: PFENet <cit.>, HDMNet <cit.>, and SegGPT <cit.>. We reproduce them following the official codes and adjust the hyperparameters to maximize performance. For PFENet and HDMNet, we train the models by cross entropy loss and Dice loss <cit.>. Due to the small objects, we adjust the downsampling modules in PFENet and HDMNet, we do not downsample the feature map. The input sizes of models are the same as SOFS. In addition, we do not mask the support features in the feature extraction of the freezing encoder like previous methods, masking leads to poor performance. We only test the model for SegGPT, and the input size is set to 448× 448, which is the same as the original setting. We do not observe the improvement if the input size is set to 512× 512.

We use ResNet-50 <cit.> and DINO v2 Base <cit.> as the encoder to extract features with freezing parameters to verify the effectiveness of the proposed method and reimplementation methods on different backbones. For ResNet-50, we use the multi-scale features from layer 2 and layer 3, and the feature processing method in <cit.> is adopted to upsample the high-level features. We do not use PSPNet <cit.> to train the base learner, which is used in the previous methods <cit.>. We find that this causes a severe overfitting problem to fit the category-specific distribution. Using the original pretrained model can alleviate this. The feature dimension C is set to 64 and C_1 is set to 64. For DINO v2 Base, we use the features from layers 5, 6, 7, 8, 9, 10. The feature dimension C is set to 256 and C_1 is set to 256. To ensure fairness, the features from the different layers of backbones for SOFS and reimplementation methods are the same. Since the backbone cannot be replaced, SegGPT is verified by ViT-Large <cit.>.





In the experiments on VISION V1 and Defect Spectrum, we train the models with 3 runs for PFENet, HDMNet, and SOFS, and then test each model with 5 runs, for a total of 15 runs. For SegGPT, we only test it with 15 runs. In the experiments of domain shifts (see Section <ref>), we test models with 10 runs. To ensure performance stability and fairness for comparison, the query/support pairs sampling by all methods are the same (the non-resizing procedure adopts the crops of the image).



 §.§ Supplemental Details for Dataset


VISION V1. VISION V1 <cit.> released in CVPR 2023 Vision-based InduStrial InspectiON workshop, is one of the largest and most diverse manufacturing datasets in terms of both image number and available annotations. VISION V1 consists of 14 objects sourced from Roboflow and <cit.>, 10k high-quality defect segmentation annotations on 4k images, spanning 44 defect types. Area proportions and pixel areas (number of pixels) for different defects are shown in Fig. <ref>. 84% of defect types have area proportions less than 0.3%, and 38% of defect types have pixel areas less than 900. These characteristics bring great challenges to defect segmentation. In the few-shot semantic segmentation, we divide all objects into 3 folds, we ensure that the distributions of area proportions of defects in each fold are roughly the same. The test objects for each fold are shown in Table <ref>.

Defect Spectrum. Defect Spectrum <cit.> relabels the data from MVTec AD <cit.>, DAGM2007 <cit.>, part of VISION V1 <cit.> and Cotton-Fabric <cit.>, ensuring that every image is accompanied by precise and diverse category annotations. It is the dataset with the largest number of defect categories so far, including in total 97 defect types. Area proportions and pixel areas (number of pixels) for different defects are shown in Fig. <ref>. 63% of defect types have area proportions less than 2%. In the few-shot semantic segmentation, we divide all objects into 3 folds, we ensure that the distributions of area proportions of defects in each fold are roughly the same. The test objects for each fold are shown in Table <ref>. Compared with the defects in VISION V1, the defects in Defect Spectrum are larger. The objects in this dataset are more under experimental conditions, , placing a single object in a noise-free background, and there is a certain gap between the defects that need to be detected in real manufacturing. Thus, we choose VISION V1 as the main experimental dataset. 





AeBAD. AeBAD <cit.> is used to detect defects in aero-engine blades. The defects of blades are under different domains, including different illuminations, backgrounds, and views. The defects contain 4 types, involving some minor defects. This dataset is used to verify the robustness of different models for different domains. In <ref>, we only test the model trained on VISION V1 <cit.> for Defect Spectrum <cit.> and AeBAD <cit.>, we verify the performance under domain shifts.



 §.§ Supplemental Experiments



The results (mean IoU) of different defect types of every object on VISION V1 are shown in Table <ref> and <ref>. SOFS achieves the best in most cases.









 §.§ Supplemental Ablation Study


The outputs of the model with and without the abnormal prior map are shown in Fig. <ref>. We can find that APM can reduce the false positive regions of missing defects.



 §.§ More Qualitative Results


We show the visualizations for different defects of a query image conditioned on different support images in Fig. <ref> and Fig. <ref>. This shows the segmentation results of SOFS for the specified defect. We show the qualitative results of every defect type on VISION V1 in Fig. <ref>, <ref>, <ref>, <ref>, <ref>, <ref>, <ref>.
























 Paper Title: What is YOLOv5: A deep look into the internal features of the popular object detector.


Distributed Adaptive Time-Varying Optimization with Global Asymptotic Convergence
    Liangze Jiang, Zheng-Guang Wu, and Lei Wang 

The authors are with College of Control Science and Engineering, Zhejiang University, Hangzhou 310027, China (e-mail: zetojiang; nashwzhg; lei.wangzju @zju.edu.cn). (Corresponding author: Zheng-Guang Wu.)
    August 5, 2024
==============================================================================================================================================================================================================================================================


§ ABSTRACT This study presents a comprehensive analysis of the YOLOv5 object detection model, examining its architecture, training methodologies, and performance. Key components, including the Cross Stage Partial backbone and Path Aggregation-Network, are explored in detail. The paper reviews the model's performance across various metrics and hardware platforms. Additionally, the study discusses the transition from Darknet to PyTorch and its impact on model development. Overall, this research provides insights into YOLOv5's capabilities and its position within the broader landscape of object detection and why it is a popular choice for constrained edge deployment scenarios.



 




§ INTRODUCTION

Computer vision is a rapidly evolving field that empowers machines <cit.> to interpret and comprehend visual information <cit.>. A critical component of this discipline is object detection <cit.>, which involves accurately identifying and locating objects within images or video sequences <cit.>. To address this challenge, various algorithmic approaches have been developed, with significant advancements achieved in recent years<cit.>.

One such groundbreaking technique is the You Only Look Once (YOLO) algorithm, introduced by Redmon et al. in 2015 <cit.>. This name reflects its distinctive approach, examining the entire image just once to identify objects and their positions. In contrast to conventional methods employing two-stage detection processes, YOLO treats object detection as a regression problem <cit.>. In the YOLO paradigm, a single convolutional neural network is employed to predict bounding boxes and class probabilities for an entire image <cit.>. This streamlined approach differs from traditional methods with more intricate pipelines.

Building upon the foundation laid by YOLO, this paper delves into the YOLOv5 architecture, a state-of-the-art object detection model that has garnered significant attention due to its exceptional performance and efficiency.





 §.§ Survey Objective
 

This study aims to evaluate the performance of the YOLOv5 object detection model in comparison to state-of-the-art alternatives. The research focuses on assessing the trade-off between model accuracy and inference speed across the various YOLOv5 variants (n, s, m, l, x). By examining these different model sizes, this study seeks to identify the optimal balance between performance metrics for diverse application requirements.

The investigation will explore the factors contributing to YOLOv5's performance gains, with particular emphasis on:


    
  * The role of PyTorch training methodologies
    
  * The impact of architectural innovations such as the CSP backbone and PA-Net neck
    
  * The effectiveness of data augmentation techniques, including mosaic augmentation
    
  * The influence of loss calculation methods and bounding box anchor generation


Additionally, the study will analyze the implications of YOLOv5's transition from the Darknet framework to PyTorch, considering aspects such as model development, deployment, and accessibility for custom object detection tasks.

By providing a comprehensive examination of YOLOv5's capabilities and innovations, this research aims to contribute to the broader understanding of advanced object detection algorithms and their practical applications in computer vision.




§ EVOLUTION OF YOLOV5

The YOLOv5 <cit.> repository emerged as an evolution of the YOLOv3 <cit.> PyTorch implementation developed by Glenn Jocher in 2020. It was released following the release of YOLOv4 <cit.>. Yolov5 gained popularity as a platform for transitioning YOLOv3 models from Darknet to PyTorch for production deployment. Recognizing the utility of this PyTorch-based approach, Ultralytics initiated a development trajectory focused on enhancing the YOLOv3 architecture and training methodologies within the PyTorch framework. This endeavor aimed to empower a broader community of developers to create and deploy custom object detectors.

This is YOLOv5 development timeline:


    
  * April 1, 2020: Initiation of development on a series of compound-scaled PyTorch models based on YOLOv3/YOLOv4 architectures.
    
  * May 27, 2020: Public release of the YOLOv5 repository, demonstrating state-of-the-art performance among existing YOLO implementations.
    
  * June 9, 2020: Integration of CSP (Cross-Stage Partial) modules, contributing to improved model speed, size, and accuracy.
    
  * June 19, 2020: Adoption of FP16 precision as default, resulting in smaller checkpoints and faster inference.
    
  * June 22, 2020: Implementation of PANet updates, including new detection heads, reduced parameters, and enhanced mAP (mean Average Precision).


Initially, the model incorporating these advancements was designated YOLOv4, aligning with the contemporaneous release of the YOLOv4 architecture within the Darknet framework. However, to avoid potential confusion and inconsistencies, it was subsequently revised to YOLOv5. While this change sparked some debate within the research community, a comparative analysis of YOLOv4 and YOLOv5 was conducted to facilitate objective evaluation <cit.>.

This study primarily focuses on the novel techniques and performance metrics introduced within the YOLOv5 framework, as detailed within the GitHub repository. It is essential to acknowledge the rapid evolution of the YOLOv5 model since its inception, which suggests ongoing research and development in this domain. As such, YOLOv5 should be considered a dynamic and evolving system rather than a static model.



§ ARCHITECTURAL FOOTPRINT OF YOLOV5

Object detection, a primary application of YOLOv5, entails the extraction of salient features from input images. These features are subsequently processed by a predictive model to localize and classify objects within the image.

The YOLO architecture introduced the end-to-end, differentiable approach to object detection by unifying the tasks of bounding box regression and object classification into a single neural network <cit.>. Fundamentally, the YOLO network comprises three core components. The backbone, a convolutional neural network, is responsible for encoding image information into feature maps at varying scales. These feature maps are then processed by the neck, a series of layers designed to integrate and refine feature representations. Finally, the head module generates predictions for object bounding boxes and class labels based on the processed features.



While there exists considerable flexibility in the architectural design of each component, YOLOv4 and YOLOv5 have significantly advanced the field by incorporating techniques from other computer vision domains. This synergistic approach has demonstrated substantial improvements in object detection performance within the YOLO framework.



 §.§ YOLOv5 Training Methods

The efficacy of an object detection system is contingent not only on its underlying architecture but also on the employed training methodologies. While architectural innovations often garner significant attention, the role of training procedures in achieving optimal performance is equally critical. There are two primary training techniques employed in YOLOv5:


    
  * Data augmentation is a pivotal component of the YOLOv5 training pipeline. By introducing diverse transformations to the training dataset, this technique enhances model robustness and generalization capabilities. Consequently, the model becomes more resilient to variations in real-world image conditions.
    
  * The Loss function is a composite metric calculated from three primary components: Generalized Intersection over Union (GIoU), objectness, and classification loss. These loss components are carefully designed to optimize mean average precision (mAP), a widely adopted evaluation metric for object detection models.




 §.§ Transition to PyTorch

YOLOv5 represents a significant advancement by transitioning the YOLO architecture from the Darknet framework to PyTorch <cit.>. The Darknet framework, primarily implemented in C, affords researchers granular control over network operations. While this level of control is advantageous for experimentation, it often hinders the rapid integration of novel research findings due to the necessity of custom gradient calculations for each new implementation.

Porting the training procedures of Darknet to PyTorch, as achieved in YOLOv3, is a complex undertaking in itself. YOLOv5 extends this effort by further optimizing and refining these procedures within the PyTorch ecosystem.



 §.§ Data augumentation

YOLOv5 incorporates data augmentation techniques within its training pipeline to enhance model robustness and generalization. During each training epoch, images are subjected to a series of augmentations through an online data loader, including:


    
  * Scaling: Adjustments to image size.
    
  * Color space manipulation: Modifications to color channels.
    
  * Mosaic augmentation: A novel technique that combines four images into four randomly sized tiles.
.

Introduced in the YOLOv3 PyTorch repository and subsequently integrated into YOLOv5, mosaic data augmentation has proven particularly effective in addressing the challenge of small object detection, a common limitation in datasets by the Common Objects in Context (COCO) object detection benchmark. By combining multiple images into a single training example, this technique exposes the model to a wider variety of object scales and spatial arrangements, thereby enhancing its ability to accurately detect smaller objects.



 §.§ Bounding Box Anchors

The YOLOv3 PyTorch repository introduced a novel approach to anchor box generation, employing K-means clustering and genetic algorithms to derive anchor box dimensions directly from the distribution of bounding boxes within a given dataset. This methodology is particularly critical for custom object detection tasks, as the scale and aspect ratios of objects often diverge significantly from those commonly found in standard datasets like COCO.

The YOLOv5 architecture predicts bounding box coordinates as offsets relative to a predefined set of anchor box dimensions. These anchor dimensions are essential for initializing the prediction process and can significantly influence the model's performance.





 §.§ Loss Calculation

The YOLOv5 loss function is a composite of three components: Binary Cross-Entropy (BCE) for class prediction and objectness, and Complete Intersection over Union (CIoU) for localization. The overall loss is computed as a weighted sum of these individual losses:


    Loss = λ_1 · L_cls + λ_2 · L_obj + λ_3 · L_loc


where L_cls, L_obj, and L_loc represent the BCE loss for class prediction, BCE loss for objectness, and CIoU loss for localization, respectively. The coefficients λ_1, λ_2, and λ_3 are hyperparameters that balance the contributions of each loss component to the overall optimization process.



 §.§ 16 Bit Floating Point Precision

The PyTorch framework offers the capability to reduce computational precision from 32-bit to 16-bit floating-point numbers during both training and inference. When applied to YOLOv5, this technique has demonstrated potential for significant acceleration in inference speed. However, it is essential to note that these performance gains is only contingent upon specific GPU architectures, notably the V100 and T4 models for Yolov5. While the limitations of hardware compatibility exist at present, ongoing developments by NVIDIA indicate a prospective expansion of this efficiency enhancement to a wider range of hardware platforms.



 §.§ CSP Backbone

Both YOLOv4 and YOLOv5 incorporate the CSP (Cross-Stage Partial) bottleneck module for feature extraction. This architectural innovation, introduced by WongKinYiu et al., addresses the issue of redundant gradient information prevalent in larger convolutional neural network backbones. By decoupling feature maps into two main parts and recombining them, the CSP module effectively reduces computational cost and model complexity without compromising performance. This efficiency enhancement is particularly advantageous for the YOLO family, where rapid inference and compact model size are paramount.

CSP models draw inspiration from the DenseNet architecture as shown in Figure <ref>, addressing the challenges inherent in deep CNNs, including the vanishing gradient problem. By fostering direct connections between layers, DenseNet aimed to enhance feature propagation, promote feature reuse, and reduce the overall number of network parameters.



The CSPResNext50 and CSPDarknet53 architectures incorporate modifications to the DenseNet structure as illustrated in Figure <ref>. Specifically, the feature maps generated by the base layer are divided into two branches. One branch is processed through a dense block, while the other is forwarded directly to the subsequent stage. This architectural modification aims to mitigate computational bottlenecks inherent to DenseNet and enhance learning by preserving an unaltered representation of the original feature map.






 §.§ PA-Net Neck

Both YOLOv4 and YOLOv5 employ the PA-Net architecture for feature aggregation <cit.>. As illustrated in Figure <ref>, each P_i represents a distinct feature layer extracted from the CSP backbone. This architectural choice is inspired by the EfficientDet object detection framework, where the BiFPN module was deemed optimal for feature integration. While the BiFPN approach serves as a benchmark, it is plausible that alternative implementations within the YOLO framework could yield further performance improvements.



It is evident that YOLOv5 builds upon the foundational research of YOLOv4 in selecting the most suitable neck architecture. YOLOv4 comprehensively evaluated a range of options including FPN, PAN, NAS-FPN, BiFPN, ASFF, and SFAM.



§ YOLOV5 MODELS

The YOLOv5 architecture encompasses five distinct models, ranging from the computationally efficient YOLOv5n to the high-precision YOLOv5x.


    
  * YOLOv5n: Designed for resource-constrained environments, YOLOv5n is the smallest and fastest model in the series. Its compact size, less than 2.5 MB in INT8 format and approximately 4 MB in FP32 format, makes it suitable for deployment on edge devices and IoT platforms. Compatibility with OpenCV DNN further enhances its utility for mobile applications.
    
  * YOLOv5s: Representing the baseline model, YOLOv5s incorporates approximately 7.2 million parameters. Its performance characteristics make it suitable for CPU-based inference tasks.
    
  * YOLOv5m: Positioned as a mid-sized model with 21.2 million parameters, YOLOv5m offers a balanced combination of speed and accuracy. This model is often considered a versatile choice for a wide range of object detection applications and datasets.
    
  * YOLOv5l: With 46.5 million parameters, YOLOv5l is designed for scenarios requiring higher precision, particularly in detecting smaller objects within images.
    
  * YOLOv5x: As the largest and most complex model in the series, YOLOv5x achieves the highest mAP among its counterparts. However, this performance comes at the cost of increased computational requirements due to its 86.7 million parameters.


A comprehensive overview of the model variants, encompassing inference speed across CPU and GPU platforms as well as the number of parameters for a 640-pixel image size, is presented in Table <ref>.







§ YOLOV5 ANNOTATION FORMAT

YOLOv5 employs a PyTorch TXT annotation format that closely resembles the YOLO Darknet TXT standard, with the addition of a YAML file specifying model configuration and class labels. To facilitate compatibility with YOLOv5, annotation data generated from various tools may require conversion. Platforms such as Roboflow offer support for multiple annotation formats and can export data directly into the YOLOv5-compatible format. Popular annotation tools including VOTT, LabelImg, and CVAT can also be utilized, with appropriate data conversion steps.



§ YOLOV5 LABELLING TOOLS

For efficient data management and annotation, Ultralytics, the developers of YOLOv5, recommends Roboflow as a compatible labeling tool <cit.>. Table <ref> provides an overview of third-party integration platforms compatible with the YOLOv5. For each platform, the table outlines its primary functionalities when integrated with YOLOv5.






§ DISCUSSION

YOLOv5 represents a significant advancement in the field of object detection, building upon the foundations laid by its predecessors while introducing novel improvements are listed below:

    
  * Architectural Advancements: YOLOv5 represents a significant advancement in object detection, building upon previous YOLO iterations. The incorporation of the CSP backbone and PA-Net neck enhances computational efficiency without compromising accuracy. The CSP module effectively addresses redundant gradient information, while the PA-Net architecture optimizes feature aggregation.
    
  * Model Versatility: The range of YOLOv5 models (n, s, m, l, x) provides flexibility for diverse hardware and application requirements. The smallest model, YOLOv5n, expands object detection capabilities to edge devices and IoT platforms.
    
  * Training Methodology Innovations: YOLOv5's emphasis on data augmentation, particularly mosaic augmentation, improves small object detection and reduces dataset size requirements. The adoption of 16-bit floating-point precision demonstrates a forward-looking approach to optimization.
    
  * Performance and Impact: YOLOv5's high mAP scores and low inference times position it as a strong contender for real-time object detection. The transition to PyTorch has democratized access to the model, fostering broader research and development.






§ CONCLUSION

YOLOv5 has emerged as a significant advancement in object detection, demonstrating a compelling balance of speed, accuracy, and user-friendliness. While the core architecture builds upon established principles, the model's implementation within the PyTorch framework represents a substantial leap forward, enhancing both development efficiency and deployment capabilities. The availability of multiple model variants tailored to diverse computational constraints expands YOLOv5's applicability across various domains from renewable energy <cit.> to quality inspection in Manufacturing <cit.>.

The model's accessibility, coupled with its compatibility with existing tools and platforms, positions YOLOv5 as a versatile solution for both research and practical applications. As the field of computer vision continues to evolve rapidly, YOLOv5 serves as a strong foundation for future advancements in real-time object detection and related tasks.



 

































unsrt  
  




 Paper Title: dopanim: A Dataset of Doppelganger Animals with Noisy Annotations from Multiple Humans.


    Learning Ordinality in Semantic Segmentation
    Rafael Cristino^1 Ricardo P. M. Cruz^1,2* Jaime S. Cardoso^1,2*
^1 Faculty of Engineering, University of Porto
^2 INESC TEC, Porto
 rpcruz@fe.up.pt, jaime.cardoso@fe.up.pt
    
===============================================================================================================================================================================

§ ABSTRACT 
    Human annotators typically provide annotated data for training machine learning models, such as neural networks. Yet, human annotations are subject to noise, impairing generalization performances. Methodological research on approaches counteracting noisy annotations requires corresponding datasets for a meaningful empirical evaluation. Consequently, we introduce a novel benchmark dataset, , consisting of about 15,750 animal images of 15 classes with ground truth labels. For approximately 10,500 of these images, 20 humans provided over 52,000 annotations with an accuracy of circa 67%. Its key attributes include (1) the challenging task of classifying pelganger als, (2) human-estimated likelihoods as annotations, and (3) annotator metadata. We benchmark well-known multi-annotator learning approaches using seven variants of this dataset and outline further evaluation use cases such as learning beyond hard class labels and active learning. Our dataset and a comprehensive codebase are publicly available to emulate the data collection process and to reproduce all empirical results.

    


§ INTRODUCTION


Supervised learning with machine learning models, such as neural networks (NNs), requires annotated data. Typically, multiple human annotators, e.g., crowdworkers <cit.>, are tasked to provide the corresponding annotations, e.g., class labels. These annotators perform differently for various reasons, including bias, fatigue, and ambiguity in interpretation <cit.>. As a result of such imperfect annotator performances, we obtain noisy annotations that can significantly degrade models' generalization performances <cit.>. Therefore, annotations are often requested from multiple annotators per data instance. The intuition is that the majority vote is a reliable estimate of the ground truth annotation (“wisdom of the crowd”). However, such an approach leads to substantially higher annotation costs and ignores the annotators' different performances. More advanced approaches have recently been proposed to improve NNs' robustness against noisy annotations. These advancements include new regularization techniques <cit.>, loss functions <cit.>, and approaches to estimate annotator performances <cit.>. The evaluation of such approaches is primarily driven by empirical research, which necessitates access to datasets that realistically reflect the noise induced by human annotators <cit.>. However, due to the often high annotation costs, the number of publicly available datasets for methodological research is relatively low. Moreover, the potential ability of certain humans to self-assess their own knowledge and uncertainty is not queried as part of the annotation campaign. Accordingly, most existing datasets are limited to classification tasks that require no expert knowledge, provide only hard class labels as annotations, and lack metadata <cit.> about the annotators. 

Motivated by the critical impact of noisy annotations in practical applications and the scarcity of corresponding datasets, we publish a novel dataset with the following profile and contributions: 
[title=: Profile and Contributions]
    
        
  (1) The  dataset[<https://doi.org/10.5281/zenodo.11479590>] features about 15,750 animal images of 15 classes, organized into four groups of pelganger als and collected together with ground truth labels from iNaturalist <cit.>. For approximately 10,500 of these images, 20 humans provided over 52,000 annotations with an accuracy of circa 67%.
        
  (2) Key attributes (cf. Fig. <ref>) include the challenging task of classifying doppelganger animals, human-estimated likelihoods per image-annotator pair, and annotator metadata.
        
  (3) The dataset's broad research scope covers noisy label learning <cit.>, multi-annotator learning <cit.>, active learning <cit.>, and learning beyond hard labels <cit.>.
        
  (4) We benchmark multi-annotator learning approaches on seven variants (annotation subsets) of our dataset to evaluate their robustness against annotation noise in diverse scenarios.
        
  (5) Three illustrative evaluation use cases regarding human-estimated likelihoods <cit.>, annotator metadata <cit.>, and annotation times in active learning <cit.> outline our datasets' potential to explore further research fields of machine learning.
        
  (6) A comprehensive codebase[<https://github.com/ies-research/multi-annotator-machine-learning>] enables emulating our data collection, reproducing all our experiments with , and includes other related datasets for a unified evaluation.
    

isarxiv


This article's remainder is structured as follows: Section <ref> discusses related datasets. Describing the data collection process of our dataset  and exemplary analyses are part of Section <ref>. We introduce variants of our dataset in Section <ref> for benchmarking multi-annotator learning approaches. Section <ref> presents further use cases of our dataset. We conclude our work in Section <ref>.
    


§ RELATED DATASETS


Learning from noisy annotations covers diverse problem settings, which mainly differ in the learning tasks and their assumptions about the annotations' origin. For example, publicly available datasets with noisy annotations from humans exist for semantic segmentation <cit.> and sequence classification tasks <cit.>. Other datasets are scraped from the web <cit.>, where noisy annotations arise due to unreliable web resources. We focus on classification tasks with annotations from multiple humans for two reasons: (1) Classification tasks are the most common research topic in learning from noisy annotations and often serve as starting points for extensions to other learning tasks. (2) Due to crowdsourcing and annotation companies, human annotators are a popular resource for annotating datasets. Along with these annotations, we can easily get information on which annotation originates from which annotator. With this scope, we use Table <ref> in the following to discuss popular and publicly available datasets regarding their task and annotation data, including  for comparison.





 §.§ Task Data


Under task data, we summarize the data essential for defining the classification task. Most datasets focus on image classification. In particular, the datasets providing noisy annotations for the popular image benchmark data , , and  are used for generic object classification. The dataset  deals with the classification of music sound files according to their genre, while  deals with the classification of the polarity of tweets. Many of these classification tasks share the characteristic that no special knowledge is required for correct classification. In contrast, our dataset  considers the challenging image classification of four groups with classes of highly similar animals. Thus, our dataset enables the investigation of learning scenarios where annotators have varying levels of domain knowledge. Except for , the number of classes in the other datasets tends to be low. This also applies to  due to the already high complexity of distinguishing the 15 animal classes. Furthermore, all datasets provide predefined splits into training and test data, but only  and  provide extra validation data to ensure reproducibility when optimizing hyperparameters. After filtering invalid images, e.g., ones showing only animal bones, the splits of  contain approximately the same number of images per class.



 §.§ Annotation Data

Under annotation data, we summarize the data related to the annotation process. The number of annotators is relatively high for almost all datasets, as these annotators are recruited via large crowdsourcing platforms such as Amazon Mechanical Turk (AMT) <cit.> and Prolific <cit.>. However, since a limited annotation budget has to be distributed among many annotators, the number of annotations per annotator is rather low. Accordingly, analyzing each annotator's annotation behavior in depth is difficult. In contrast, the annotations of  originate from fewer annotators, each of whom has provided many annotations via LabelStudio <cit.>. The partially low overall annotation accuracies for the datasets and the large differences between the annotators' individual accuracies demonstrate the need for multi-annotator learning techniques. Beyond hard class labels, more informative annotation types can also be requested. In particular, soft class labels capture the annotators' subjective uncertainties regarding their decisions. For , the probabilities for the two most probable class labels and any class label to which the image does definitely not belong are available. Our dataset  provides soft class labels, where the annotators could distribute unnormalized likelihood scores across all class labels to reflect their uncertainties. Other important data can be collected in addition to the annotations when annotating. This includes metadata about the annotators <cit.>, such as self-assessed motivation, and their annotation times. While annotation times are provided by some other datasets, detailed annotator metadata is only provided by . 
 
    


§ : A DATASET OF DOPPELGANGER ANIMALS


This section describes the task and annotation data collection. Further details, including ethical considerations, are given in the supplementary material and the codebase to emulate the data collection.



 §.§ Collection and Analysis of Task Data

Our dataset  targets a classification task with images of animal species with groups of highly similar appearances, which we call doppelgangers. Therefore, accurate annotations require domain knowledge and a high level of attention. The images originate from iNaturalist <cit.>, a citizen science project and online social network of biologists and citizen scientists who contribute biodiversity observations across the globe. 
[14]r0.4
  

  

    
    < g r a p h i c s >

  

  t-SNE of validation images' features from a    fine-tuned on .
  

iNaturalist is particularly suitable as an image source because it contains a vast and diverse collection of high-quality images contributed by users worldwide. Each observation we collected from iNaturalist is licensed as CC0, CC-BY, or CC-BY-NC and includes metadata such as location and time. The associated images span 15 classes across four animal groups, each containing doppelganger animals. Research-grade images, validated by multiple users, ensure reliable ground truth class labels and, thus, the dataset's reliability for evaluation. We demonstrate the animals' doppelganger characteristics within a group by fine-tuning a DINOv2 vision transformer (ViT) <cit.> on training data with ground truth class labels. The learned features are reduced to a two-dimensional space via t-distributed stochastic neighborhood embedding (t-SNE) <cit.>, depicting validation animal images as markers in Figure <ref>. While the four groups (marker shapes) are clearly distinguishable, animal classes within a group (marker colors) exhibit high similarities.



 §.§ Collection and Analysis of Annotation Data

We organized the annotation data collection into several steps. Initially, student assistants were recruited as annotators. Each annotator received a basic tutorial on using LabelStudio <cit.> as the annotation platform, including an overview of the annotation interface and an example image for each of the 15 animal classes. Additionally, certain annotators received advanced tutorials on specific animal groups to ensure different levels of expertise. After studying the tutorials, annotators completed a pre-questionnaire with self-assessment and technical questions about wildlife. The actual annotation tasks involved assigning unnormalized label likelihoods to images, as shown in Fig. <ref>. Once the annotators had completed their annotation tasks, they filled out a post-questionnaire with self-assessment questions about the difficulties they encountered during the annotation process.


[15]r0.4
  

    
    < g r a p h i c s >

  

  Confusion matrix across all human top-label predictions.
  

The collected annotation data comprises four main components: (1) tutorials, (2) pre-questionnaire, (3) post-questionnaire, and (4) individual image annotations. Components (1)-(3) allow us to extract annotator metadata, which captures task-related information about the annotators, e.g., the interest in wildlife. In addition to the assigned unnormalized likelihoods, component (4) includes the required annotation times and timestamps. All these annotation data are detailed in the supplementary material and can be used to evaluate various learning scenarios (cf. Section <ref> for corresponding use cases). For example, we can determine the top-label predictions from the likelihoods to compute the confusion matrix in Fig. <ref>. We can see that animal classes are mainly confused within a group of doppelganger animals.


    


§ BENCHMARK: MULTI-ANNOTATOR LEARNING


This section presents a benchmark of common approaches in multi-annotator learning <cit.>, also called learning from crowds <cit.>. We briefly outline this research area's foundations before presenting seven variants of  as the basis for the experimental setup, empirical results, and future research. Our supplementary material further details this benchmark, e.g., by listing computational resources.



 §.§ Foundations

Multi-annotator learning approaches typically assume independently working annotators and require knowing which annotation originates from which annotator. This information allows them to estimate annotators' individual performances for correcting noisy class labels. Major differences in the approaches arise in their assumptions about these performances and their training procedures <cit.>. 

Annotation Performance Assumptions. The simplest assumption is that an annotator's performance is constant across all classes and instances, often represented by a single accuracy score per annotator <cit.>. However, this assumption is unrealistic, e.g., due to varying difficulty levels across different classes and instances. Therefore, performance is often modeled with class dependency, typically by estimating a confusion matrix for each annotator <cit.>. This matrix captures the probability of an annotator correctly or incorrectly annotating each class. Some approaches also incorporate instance dependency, which considers the variability in annotator performance based on specific instances <cit.>. For example, annotators may perform better in certain regions of the feature space. However, this most realistic assumption comes at the cost of increased training complexity.

Training Procedures. Training procedures are often divided into one-stage and two-stage procedures <cit.>. Two-stage procedures aggregate multiple class labels per instance as estimates of the ground truth class labels in the first stage. These aggregated labels are then used for standard supervised learning in the second stage. The label aggregation is implemented via ground truth inference algorithms <cit.>. The simplest algorithm is majority voting, specifying an instance's class label as the one with the most annotator votes. Thereby, majority voting assumes annotators have equal performance <cit.>. Advanced ground truth inference algorithms <cit.> drop this naive assumption by estimating annotators' performances for label aggregation.  All these algorithms typically require multiple class labels per instance <cit.>, leading to high annotation costs. One-stage training procedures overcome this requirement, enabling learning with just one class label per instance. This involves training two models: a classification model and an annotator performance model <cit.>. Early one-step procedures leverage the expectation-maximization (EM) algorithm, where the E-step estimates the ground truth labels as latent variables, and the M-step updates the models based on these estimates <cit.>. State-of-the-art procedures use NN-based end-to-end systems, coupling both models' outputs into a single loss function for simultaneous training <cit.>.




 §.§ Dataset Variants

We create seven dataset variants (annotation subsets) of  to test different learning scenarios. Two critical variables in multi-annotator learning are annotator performance and the number of annotations per instance <cit.>. We simulate varying annotator performance levels by either omly selecting annotations per instance or by selecting the  (false if available) annotations per instance. Further, we control the number of annotations per instance, testing scenarios with very few ( or ), a variable number (), and many () annotations per instance. Table <ref> summarizes the statistics of the seven variants. For future research, dataset users are free to create additional variants.





 §.§ Experimental Setup

Approaches. We consider state-of-the-art approaches from the literature. Thus, we focus on one-step end-to-end approaches for training NNs, which estimate class-dependent or instance-dependent annotator performances. Further, we evaluate the training with the ground truth class labels as the upper baseline () and the training with the majority vote class labels as the lower baseline (). Table <ref> lists all eleven approaches in its header. For transparency, we note that the approaches  <cit.> and  <cit.> were proposed by our research group in previous works.

Evaluation Scores.  For quantitative evaluation, we employ three scores suitable for balanced classification problems. Accuracy evaluated on the test data is the most common measure of generalization performance. Since many applications require more than just the actual class prediction, we also assess the quality of the predicted class probabilities. Specifically, we employ the Brier score <cit.> as a proper scoring rule and the top-label calibration error <cit.> with a more intuitive interpretation.

[12]r0.33
        

        
        1.1
        Hyperparameters.
        
        
            1|c|Hyperparameter     Value 

            2|c|datasetcolorArchitecture 

            backbone     DINOv2 ViT-S/14

            classification head      MLP 

            2|c|datasetcolorTraining 

            optimizer     RAdam 

            learning rate scheduler     cosine annealing

            number of epochs     50 

            initial learning rate     0.001

            batch size     32 

            weight decay     0

            dropout rate     0.5

            

Architecture. With the advance of self-supervised learning <cit.>, numerous pre-trained model architectures for image data are available, making training from scratch necessary only in special applications. Thus, we use a pre-trained DINOv2 ViT-S/14 <cit.> as our backbone. Training of the multi-annotator learning approaches is then implemented by fine-tuning the backbone's classification head in the form of a multi-layer perceptron (MLP) with 128 hidden neurons, batch normalization <cit.>, dropout <cit.>, and rectified linear units (ReLU) <cit.> as activation function.

Training. We employ RAdam <cit.> as the optimizer across all dataset variants and multi-annotator learning approaches. Further, we schedule the learning rate over 50 training epochs via cosine annealing <cit.>. Concrete values for the other hyperparameters, i.e., initial learning rate, batch size, weight decay, and dropout rate, are determined by optimizing the validation accuracy of the upper baseline  and are reported in Table <ref>. Hyperparameters specific to each approach are defined according to the authors' recommendations. This way, we aim to avoid biasing results in favor of or against any particular approach, allowing for a fair empirical comparison <cit.>. Each training is repeated ten times with different random initializations of the NNs' parameters per dataset variant and approach. All documented results are reported as means and standard deviations over the ten repetitions of the respective experiment.



 §.§ Empirical Results

[20]r0.33
    

    

        
    < g r a p h i c s >

    

    Mean ranks (↓) across the seven dataset variants.
    

Table <ref> reports the individual results for all approaches, dataset variants, and evaluation scores. A more compact overview of these results (excluding the upper baseline ) is given in the form of a ranking by Fig. <ref>. Lower ranks indicate better evaluation scores. As expected, the tabular results confirm that training with the ground truth class labels () is superior across all dataset variants and evaluation scores. Training with the majority vote class labels () leads to inferior accuracy results, thus confirming the benefit of multi-annotator learning approaches. This is particularly prominent for scenarios with low annotator performances, as the results for  variants show. In contrast, the Brier scores and top-label calibration errors of  are partially competitive, suggesting room for improving the multi-annotator learning approaches' calibration. Approaches modeling instance-dependent annotator performances are not consistently better than approaches modeling only class-dependent annotator performances, likely due to the more complex training of annotator performance models. Overall, the most superior approach is  <cit.> using a  <cit.> extension, followed by  <cit.> with a regularized loss function. 

[left=0.1cm, right=0.1cm, top=0.0cm, bottom=0.0cm]
    Takeaway: The empirical results on the seven variants of  demonstrate the benefit of using multi-annotator learning approaches to counteract annotation noise.



    


§ FURTHER USE CASES


This section discusses three illustrative evaluation use cases to demonstrate our dataset's potential for exploring further research areas. More detailed analyses are given in the supplementary material.



 §.§ Beyond Hard Class Labels

Foundations.  Instead of forcing hard decisions on one class label, human annotators can assign likelihoods to express their subjective uncertainty about an annotation decision.

Research Question. Are the human-estimated likelihoods reliable, and can they act as weights in a majority voting to improve the performance of ?

Experimental Setup. We qualitatively evaluate the human-estimated likelihoods across all annotators through a reliability diagram <cit.> by normalizing them to obtain top-label probabilities. Such a reliability diagram depicts the annotation accuracy (y-axis) as a function of the top-label probability (x-axis) in the form of a calibration curve. The latter quantity can also be interpreted as the annotators' confidences, whose distribution can be visualized through a histogram. Following the setup of Section <ref>, we perform a quantitative evaluation by leveraging the likelihoods in a weighted majority voting for training  on our seven dataset variants.

Empirical Results. Figure <ref> displays the obtained reliability diagram.  Although the annotators are very confident in most of their decisions, there are also various annotations with low confidence. The general trend of the observed calibration curve shows that a higher top-label probability implies a higher accuracy. Yet, the annotators tend to overestimate their accuracies for annotations with very high confidences. Table <ref> lists the accuracy gains of training  with likelihoods as weights for majority voting. We observe positive gains for six out of seven variants. The likelihoods are particularly helpful for settings with many false annotations, as shown by the substantial gains for the three  variants.

[left=0.1cm, right=0.1cm, top=0.0cm, bottom=0.0cm]
    Takeaway: Human-estimated likelihoods as part of  can be leveraged to counteract annotation noise.




 §.§ Annotator Metadata

Foundations. Annotator metadata <cit.> describes task-related properties of the annotators. Ideally, this information can be collected cost-effectively and is connected to annotator performance.

Research Question. Can multi-annotator learning approaches benefit from annotator metadata? 

Experimental Setup. Before and after the annotation campaign, each annotator completed a questionnaire in which only task-related data was collected. Accordingly, the metadata does not reveal an annotator's identity. We include further information about absolved tutorials and compute annotation statistics for each annotator, e.g., mean annotation time and an annotator's consistency when processing the same images multiple times. All this information is summarized as a vector of metadata features per annotator. We analyze the potential benefit of these features through correlation analysis and by integrating the metadata into the training of multi-annotator learning approaches. We follow the setup of Section <ref> and employ  because it can process the metadata as vectorial inputs.

Empirical Results. Figure <ref> presents Spearman's rank correlation coefficients <cit.> between observed annotator accuracies and selected annotator metadata features. Notably, there is a strong correlation between annotator accuracy and both the accuracy of answers to technical wildlife questions and the consistency of annotations across identical images. Table <ref> lists the accuracy gains of training  with additional annotator metadata. There are positive gains for five out of seven variants.

[left=0.1cm, right=0.1cm, top=0.0cm, bottom=0.0cm]
    Takeaway: Annotator metadata as part of  can be leveraged to improve the training of multi-annotator learning approaches.





 §.§ Annotation Times in Active Learning

Foundations. Active learning <cit.> strategies aim to reduce the annotation cost while maximizing the generalization performance by selecting only the most useful instances for annotation. The most common strategy, uncertainty sampling <cit.>, selects instances with the highest model uncertainty.

Research Question. Do instances selected by uncertainty sampling require more time for annotation?

Experimental Setup.  We compare uncertainty sampling to random sampling as a baseline strategy. For simplicity, we use the ground truth labels to fine-tune a linear layer as the classification head of the pre-trained DINOv2 ViT-S/14 model <cit.>. At the start of active learning, we randomly annotate 100 images. Subsequently, the respective strategy selects 100 new images for annotation in each following iteration. We repeat this experiment ten times and report means and standard deviations.

Empirical Results. The histogram in Fig. <ref> indicates high variance in annotation times, which is also reflected in the active learning results shown in Fig. <ref>. Uncertainty sampling achieves higher accuracies (y-axis) with the same number of annotations (x-axis) than random sampling. However, the mean annotation time for images selected by uncertainty sampling is almost 1s longer.


[left=0.1cm, right=0.1cm, top=0.0cm, bottom=0.0cm]
    Takeaway: With , we demonstrated that uncertainty sampling selects images with higher annotation times.


.3
        
    < g r a p h i c s >

        figureCalibration plot of human top-label probabilities. 
        
        
        1.2
        tableProbabilistic Labels
        
        
            1|c|Resultsplaceholder     xxxxxxxx 

            2|c|datasetcolor Test Accuracy Gains (+ [%]) 

                  +9.0_± 0.8

                  +8.4_± 0.5

                  +5.3_± 0.5

                 -0.7_± 0.6

                  +2.2_± 0.7

                  +1.5_± 0.3

                  +0.8_± 0.3

            
.3
        
    < g r a p h i c s >

        figureCorrelation of annotator metadata and accuracies.    
        
        
        1.2
        tableAnnotator Metadata
        
        
            1|c|Resultsplaceholder     xxxxxx 

            2|c|datasetcolor Test Accuracy Gains (+ [%]) 

                  +1.1_± 1.9

                 -0.2_± 0.8

                  +0.4_± 0.8

                 +0.0_± 0.3

                  +0.3_± 0.5

                  +0.3_± 0.4

                  +0.3_± 0.4

            
.3
        

        
    < g r a p h i c s >

        figureHistogram of all recorded annotation times.        
        

        

        

        
    < g r a p h i c s >

        figureActive learning curves of accuracy and times.    
        

    


§ CONCLUSION AND LIMITATIONS


Conclusion. We introduced the  dataset containing images with ground truth labels of animal species with groups of similar appearance (pelganger als). In a comprehensive annotation campaign, error-prone humans annotated this dataset using likelihoods to express their subjective uncertainty. These annotations are supplemented with annotator metadata <cit.> containing task-related information obtained from questionnaires, tutorials, and annotation statistics. A benchmark study evaluated the benefit of multi-annotator learning approaches <cit.> for seven variants of . Moreover, three use cases demonstrated 's potential for further research. Data and codebase, including multi-annotator learning approaches, backbones, experiments, and evaluation protocols, are publicly accessible, facilitating methodological research in various machine learning fields.

Limitations. Due to the high costs of annotations,  is a small-scale dataset. To test scalability, significantly more images and animal classes must be included and annotated. For this purpose, our codebase for collecting  using iNaturalist <cit.> and LabelStudio <cit.> can be easily adapted. Additionally, our benchmark of multi-annotator learning approaches is limited to the seven variants of  and DINOv2 ViT-S/14 <cit.> as the backbone. However, our codebase includes other backbones and related datasets with noisy annotations from multiple humans, allowing for future benchmark extension. Such an extension may also include approaches for learning from noisy class labels <cit.>, which do not rely on the information which class label originates from which annotator.
    
    
    
        This work was funded by the ALDeep and CIL projects at the University of Kassel. Moreover, we thank Franz Götz-Hahn for his insightful comments on improving our annotation campaign. Finally, we thank the iNaturalist <cit.> community for their many observations that help explore our nature's biodiversity and our annotators for their dedicated efforts in making the annotation campaign via LabelStudio <cit.> possible.
    
    
    
    plainnat
    
    
    
    


§ CHECKLIST





  * For all authors...

  
  * Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
    The main claims are presented in the box of Section <ref>.
  
  * Did you describe the limitations of your work?
    We describe the limitations regarding our dataset and associated experiments in Section <ref>.
  
  * Did you discuss any potential negative societal impacts of your work?
    In the supplementary material, we discuss the potential societal issues and impacts of employing humans as annotators and training with noisy annotations. Moreover, we describe our actions to avoid such negative impacts on our annotators during the annotation campaign.
  
  * Have you read the ethics review guidelines and ensured that your paper conforms to them?
    We carefully studied the ethics review and discuss its different aspects in the context of our work in the supplementary material.



  * If you are including theoretical results...

  
  * Did you state the full set of assumptions of all theoretical results?
    
	
  * Did you include complete proofs of all theoretical results?
    



  * If you ran experiments (e.g. for benchmarks)...

  
  * Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
    Section <ref> lists the links to the data and code repository, which provide further technical details (e.g., step-by-step guides) on downloading the data and reproducing all experimental results.
  
  * Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
    In addition to Section <ref>, the supplementary material and our codebase overview the experimental setup, including how the hyperparameters in Table <ref> were chosen.
	
  * Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
    All results in Tables <ref>, <ref>, <ref>, and Fig. <ref> refer to the mean and standard deviation obtained from ten repetitions of the respective experiment.
	
  * Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? 
    A dedicated section in the supplementary material provides the required information about the used computing resources of our lab's proprietary cluster and associated execution times.  



  * If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...

  
  * If your work uses existing assets, did you cite the creators?
    We reference assets directly important for the annotation campaign and the experimental setup in the main paper, e.g., the annotation platform and backbone, while the remaining assets, such as helpful code packages, are referenced in the supplementary material.
  
  * Did you mention the license of the assets?
    Each asset with an available license is mentioned in the supplementary material.
  
  * Did you include any new assets either in the supplemental material or as a URL?
    New assets, to which our released dataset and code belong, are linked in Section <ref>.
  
  * Did you discuss whether and how consent was obtained from people whose data you're using/curating?
    In Section <ref> and the supplementary, we describe how we ensured to collect only images and image metadata with proper licenses (CC0, CC-BY, CC-BY-NC) from iNaturalist <cit.>. The collected annotation data is published for research purposes (CC-BY-NC 4.0) with the consent of all annotators.
  
  * Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
    We obey national and international (European Union) law, in particular, the General Data Protection Regulation (GDPR) <cit.>, to make revealing an annotator's identity impossible. Details are given in the supplementary material.



  * If you used crowdsourcing or conducted research with human subjects...

  
  * Did you include the full text of instructions given to participants and screenshots, if applicable?
    We discuss all tutorials, questionnaires, and annotation interfaces in the supplementary material. Moreover, we provide the associated material to reproduce the image and annotation collection as part of our codebase.
  
  * Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
     No IRB approval was necessary because the participants were employees of our institution, and they annotated the images as part of their employment. To minimize any potential risks, we complied with the GDPR <cit.>, which gives the participants the right to reject the annotation work at any time and ensures that no participant can be identified with the publication of the dataset.
  
  * Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
    Since the annotators were student assistants at a German university, the minimum wage law was adhered to. Specific numbers, varying by educational qualification, are provided in the supplementary material.





    
    isarxiv
    

§ OVERVIEW


The supplementary material provides additional information beyond the primary resources of our dataset , which are:

    
  * the main paper presenting our core findings,
    
  * the dataset on Zenodo[<https://doi.org/10.5281/zenodo.11479590>] offering direct access to the data,
    
  * and the codebase on GitHub[<https://github.com/ies-research/multi-annotator-machine-learning>] ensuring reproducibility and ficilating experimentation.

We structure this supplementary material as follows: Section <ref> presents the datasheet of the  dataset based on a standard questionnaire. Assets used within this work are listed in Section <ref>. We discuss our dataset in the societal context of collecting data with the help of humans in Section <ref>. 
Further related datasets with noisy annotations, which did not fit our main paper's focus, are detailed by Section <ref>. 
Finally, we provide more information about the empirical evaluation in Section <ref>.
        


§ DATASHEET


In this section, we present the datasheet of our dataset  by completing the questionnaire developed by <cit.>. The purpose of such a datasheet is to provide detailed documentation that describes the various aspects of a dataset, including its motivation, composition, collection process, preprocessing, uses, distribution, and maintenance.



 §.§ Motivation


For what purpose was the dataset created? Was there a specific task in mind? Was there a specific gap that needed to be filled?

The  dataset was created to advance methodological research in machine learning with noisy annotations from multiple annotators. Next to multi-annotator learning <cit.>, also referred to as learning from crowds <cit.>,  concrete use cases range from learning with noisy class labels <cit.> and learning beyond hard class labels <cit.> to active learning with error-prone annotators <cit.>.

Who created the dataset (e.g., which team, research group) and on behalf of which entity (e.g., company, institution, organization)? If there is an associated grant, please provide the name of the grantor and the grant name and number.

The dataset was created by the researchers Marek Herde, Denis Huseljic, and Lukas Rauch of the Intelligent Embedded Systems (IES) group at the University of Kassel under the supervision of Bernhard Sick. 

Who funded the creation of the dataset? 

This work was funded by the ALDeep and CIL projects, with the University of Kassel as the exclusive project sponsor. 

Any other comments?

We refer to the main paper for a more detailed description of our dataset's motivation and objectives.



 §.§ Composition

What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description.

The dataset comprises instances in the form of animal images, which are supplemented by task data (cf. Table <ref>), annotation data (cf. Table <ref>), and annotator metadata (cf. Table <ref>).





How many instances are there in total (of each type, if appropriate)?

The dataset includes around 15,731 animal images (records of task data). Out of these, 10,484 images have been annotated by 20 human annotators (records of annotator metadata), resulting in a total of  52,795 annotations (records of annotator metadata).

Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable)

The dataset comprises a sample of images taken from iNaturalist <cit.>, focusing on 15 animal classes belonging to four groups of doppelganger animals that are challenging to differentiate. As a result, these images are suitable for studying challenging classification tasks (requiring expertise for correct annotation) in the field of learning from noisy annotations <cit.>.

What data does each instance consist of? “Raw” data (e.g., unprocessed text or images) or features? In either case, please provide a description

Each image is a  file named according to the unique observation identifier on iNaturalist. The task data contains additional information for each image, e.g., the location (cf. Table <ref>). A record in the annotation data consists next to the likelihoods of further information about the annotation process, e.g., annotation time (cf. Table <ref>). Task-related information about the annotators, e.g., self-estimated accuracy, acquired via questionnaires form a record in our annotator metadata (cf. Table <ref>).



Is there a label or target associated with each instance? If so, please provide a description.

Each image has a single ground truth class label obtained from iNaturalist (cf.  in Table <ref>). Furthermore, about five annotations in the form of unnormalized class-label likelihoods estimated by human annotators are assigned to each image (cf.  in Table <ref>). 

Is any information missing from individual instances? If so, please provide a description, explaining why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text.

Aside from the fact that not every annotator processed every image due to the high annotation workload and limited budget, the dataset is complete.



Are relationships between individual instances made explicit (e.g., users’ movie ratings, social network links)? If so, please describe how these relationships are made explicit.

The relationships between images, task data, annotation data, and annotator metadata are made explicit by using corresponding identifiers. More concretely, images are named according to the  in the task data (cf. Table <ref>). This unique identifier is also part of the annotation data (cf. Table <ref>) to map annotations to images. Likewise, the  in the annotator metadata (cf. Table <ref>) creates a mapping to the annotation data.

Are there recommended data splits (e.g., training, development/validation, testing)? If so, please provide a description of these splits, explaining the rationale behind them.

3b To ensure reproducibility, we provide fixed training, validation, and test splits (cf. data field  in Table <ref>), each with a balanced class distribution.

Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide a description.

The annotations in our dataset were collected to capture human annotation noise. Additionally, each annotator processed several images twice to assess the consistency of their annotations.

Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? If it links to or relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) are there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to a dataset consumer? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, as appropriate.

The dataset is self-contained and only provides links to the observations and images for reference.

Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals’ non-public communications)? If so, please provide a description.

5e No (cf. Section <ref> for further explanation). 

Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? 

iNaturalist aims to capture the diversity of all wildlife, which includes images of deceased animals. This is also true for a few instances in our dataset.  Annotators were given a trigger warning and could withdraw from the annotation campaign anytime.

Does the dataset identify any subpopulations (e.g., by age, gender) If so, please describe how these subpopulations are identified and provide a description of their respective distributions within the dataset.

5e No (cf. Section <ref> for further explanation). 

Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset?  If so, please describe how

5e No (cf. Section <ref> for further explanation). 

Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals race or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? If so, please provide a description.

5e No (cf. Section <ref> for further explanation). 


Any other comments?

None.



 §.§ Collection Process

How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)? If the data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verified? If so, please describe how.

Our data was collected in two phases. In the first phase, we downloaded observations with corresponding images from iNaturalist. These data were then split into training, validation, and test sets. The training data was further divided into ten batches of annotation tasks, which were manually assigned to different annotators. Before starting the annotation process, each annotator completed one or more tutorials with annotation instructions and information on distinguishing between the 15 animal classes. Additionally, each annotator filled out a pre-questionnaire (cf. Fig. <ref>). During the annotation tasks, annotators could assign likelihoods and mark invalid images (cf. Fig. <ref>). After processing all assigned annotations, a post-questionnaire (cf. Fig. <ref>) was filled out by each annotator.






What mechanisms or procedures were used to collect the data (e.g., hardware apparatuses or sensors, manual human curation, software programs, software APIs)? How were these mechanisms or procedures validated?

We used pyinaturalist <cit.> as a Python interface for the iNaturalist API to download the observations and images. We organized the annotation campaign using Label Studio <cit.>, with a community edition instance set up for each annotator on an internal workstation of our university. The correctness of all implementations was ensured through preliminary tests. Code to emulate our data collection is available at our  repository.

If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)?

The dataset is a small random sample of 15 animal classes downloaded from iNaturalist.

Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)?

We managed the collection of this dataset and employed 20 student assistants making valuable contributions as annotators. Section <ref> provides further details regarding their compensation.



Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of new news articles)? If not, please describe the timeframe in which the data associated with the instances was created.

Downloading the observations and images took approximately two to three days, while the annotation campaign spanned over four months.

Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)?

We collected all data from the annotators using Label Studio.

Were the individuals in question notified about the data collection? If so, please describe (or show with screenshots or other information) how notice was provided, and provide a link or other access point to, or otherwise reproduce, the exact language of the notification itself.

Due to the applicable licenses (cf. Table <ref>), no notification was sent to the iNaturalist users as data creators of observations and images. However, the annotators were explicitly notified about the usage of their annotation data (cf. Section <ref>).

Did the individuals in question consent to the collection and use of their data? If so, please describe (or show with screenshots or other information) how consent was requested and provided, and provide a link or other access point to, or otherwise reproduce, the exact language to which the individuals consented.

4d No consent was required to collect data from iNaturalist due to the applicable licenses (cf. Table <ref>). However, the annotators were explicitly asked for their consent (cf. Section <ref>).

If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses? If so, please provide a description, as well as a link or other access point to the mechanism (if appropriate).

Yes (cf. Section <ref>).

Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted? If so, please provide a description of this analysis, including the outcomes, as well as a link or other access point to any supporting documentation.

No data protection impact analysis was conducted; however, all annotators were informed beforehand about the potential tasks and outcomes. Additionally, all steps complied with the General Data Protection Regulation (GDPR) <cit.>. More information is provided in Section <ref>.

Any other comments?

None.



 §.§ Preprocessing/cleaning/labeling

Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. If not, you may skip the remaining questions in this section.

After downloading the observations and images, we filtered with the help of the annotators (cf. Fig. <ref>) invalid images showing no animals or depicting at least two animals of the 15 classes (multi-label).

Was the “raw” data saved in addition to the preprocessed/cleaned/labeled data (e.g., to support unanticipated future uses)? If so, please provide a link or other access point to the “raw” data.

The invalid images were manually reviewed by us, the dataset managers, to ensure they were correctly tagged as invalid. As a result, there is no value in retaining these images. All remaining data is available as part of our dataset.

Is the software that was used to preprocess/clean/label the data available? If so, please provide a link or other access point.

All tools employed to collect the dataset are freely available and are described in Section <ref>.

Any other comments? 

None



 §.§ Uses

Has the dataset been used for any tasks already? If so, please provide a description.

Except for the presented usages in the main paper, the dataset has not been employed for further tasks until now. 

Is there a repository that links to any or all papers or systems that use the dataset? If so, please provide a link or other access point.

Since the dataset is new, it has not yet been used in other papers. However, we will maintain a list in our  repository that provides a corresponding overview with brief descriptions.

What (other) tasks could the dataset be used for?

As discussed in the main paper, our multipurpose dataset can be used to research various learning tasks. These tasks include, but are not limited to:


    
  * standard image classification,
    
  * learning beyond hard labels <cit.>,
    
  * learning from noisy annotations <cit.>,
    
  * multi-annotator learning <cit.>,
    
  * annotation aggregation <cit.>,
    
  * and active learning <cit.>.


Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? For example, is there anything that a dataset consumer might need to know to avoid uses that could result in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other risks or harms (e.g., legal risks, financial harms)? If so, please provide a description. Is there anything a dataset consumer could do to mitigate these risks or harms?

The observations and associated images used as task data were carefully curated from iNaturalist under their respective licenses. The licensing information is provided by iNaturalist users and cannot be verified with absolute certainty. If any potential license or copyright violations (cf. Section <ref>) are discovered, we will correct them promptly. In such a case, dataset consumers will be informed via  and  to adjust their data accordingly.

Are there tasks for which the dataset should not be used? If so, please provide a description.

The dataset must be used exclusively for non-commercial research tasks. Moreover, the annotator metadata has been carefully designed and is not intended to motivate the inclusion of sensitive and protected personal data in similar collections.

Any other comments? 

Dataset consumers must ensure their dataset's usage complies with all licenses, laws, regulations, and ethical guidelines (cf. Section <ref>).



 §.§ Distribution

Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide a description.

The dataset is accessible to everyone.

How will the dataset be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have a digital object identifier (DOI)?

The dataset is available through  with a respective DOI. Code to load and experiment with the dataset is provided via .

When will the dataset be distributed? 

The dataset is already publicly available.

Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If so, please describe this license and/or ToU, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions.

4b, c Section <ref> and Table <ref> provide an overview of our dataset's licenses.

Have any third parties imposed IP-based or other restrictions on the data associated with the instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms, as well as any fees associated with these restrictions.

No.

Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any supporting documentation.

No.

Any other comments? 

None.



 §.§ Maintenance


How can the owner/curator/manager of the dataset be contacted (e.g., email address)? 

If questions or issues are relevant to other potential dataset consumers, we ask to create a corresponding issue at our  repository (codebase). In all other cases, the dataset consumers can contact the dataset collectors via the e-mail mailto:marek.herde@uni-kassel.de.

Is there an erratum? If so, please provide a link or other access point.

No errors have been found so far.

Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? If so, please describe how often, by whom, and how updates will be communicated to dataset consumers (e.g., mailing list, GitHub)?

Currently, no extensions are planned to improve the comparability of future evaluation results on our dataset. However, if any errors are discovered, they will be corrected, and the dataset will be updated with a new version number on . The change will also be announced in our  repository.

If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were the individuals in question told that their data would be retained for a fixed period of time and then deleted)? If so, please describe these limits and explain how they will be enforced.

In compliance with the GDPR (cf. Section <ref>), all annotators were asked for their consent to use their contributions for scientific purposes. Additionally, all collected metadata from the annotators has been fully anonymized, so there are no restrictions on data retention.

Will older versions of the dataset continue to be supported/hosted/maintained? If so, please describe how. If not, please describe how its obsolescence will be communicated to dataset consumers.

All dataset versions will be available on .

If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? If so, please provide a description. Will these contributions be validated/verified? If so, please describe how. If not, why not? Is there a process for communicating/distributing these contributions to dataset consumers? If so, please provide a description.

We warmly welcome scientific contributions from the community in the context of our dataset. Contributors are encouraged to create an issue on  to discuss potential implementations. Code contributions can also be integrated via pull requests. For correctness and transparency, we and the community will publicly discuss and verify each contribution.

Any other comments? 

None.
        


§ ASSETS AND LICENSES


4a, b This work integrates various well-established assets, i.e., platforms, code, and models, to facilitate the creation of our dataset  and the associated empirical evaluation. In this way, we maximize the correctness of our results, including improved reproducibility and access to other users.

For the collection and publication of the task data, annotation data, and annotator metadata, we mostly relied on the following assets:

    
  * iNaturalist <cit.> (platform, individual license) was the source for collecting the task data, e.g., images and their ground truth class labels.
    
  * pyinaturalist <cit.> (code, MIT license) implements a Pythonic interface to the iNaturalist API and was used for downloading and filtering observations.
    
  * LabelStudio <cit.> (platform, Apache 2.0 license) is an open-source tool for annotating various types of data and its community edition served as annotation platform for our work.
    
  * Zenodo <cit.> (platform, GPL-2.0 license) is an open platform for sharing and preserving research outputs of various formats, including our dataset .


For the empirical evaluation consisting of a benchmark and case studies on seven variants of our dataset , we mostly relied on the following assets:

    
  * PyTorch <cit.> (code, individual license) and PyTorch Lightning <cit.> (code, Apache-2.0 license) are central Python packages for implementing and training deep learning models, corresponding to multi-annotator learning approaches in our case.
    
  * DINOv2 <cit.> (model, Apache-2.0 license) offers multiple self-supervised learning models, of which we employed the ViT-S/14.
    
  * Hydra <cit.> (code, MIT license) is a framework to configure complex applications corresponding to our work's experiments with their hyperparameters.
    
  * MLFlow <cit.> (code, Apache-2.0 license) supports the organization of machine learning projects and assisted us with the logging and reading of the empirical results.
    
  * Scikit-learn <cit.> (code, BSD-3-Clause license) is a famous machine learning framework, of which we used pre-processing techniques and logistic regression.
    
  * Scikit-activeml <cit.> (code, BSD-3-Clause license) is a library of active learning algorithms employed by us for the use case on annotation times in active learning.


Further assets, such as NumPy <cit.> and Pandas <cit.> as standard Python packages in scientific computing, are included as requirements of working with our codebase.

4b, c Table <ref> provides an overview of our  dataset's licenses. The task data retains the licenses from iNaturalist, while the annotation data and annotator metadata are collected by us and distributed under the license CC-BY-NC 4.0. Our codebase on  to load and experiment with the dataset is available under the BSD-3-Clause license. As authors, we bear full responsibility for removing parts of our dataset or codebase or withdrawing our paper if confirmed violations of licensing agreements, intellectual property rights, or privacy rights are identified. In such a case, dataset consumers will be informed via  and  to adjust their data accordingly. Moreover, dataset consumers are responsible for ensuring their dataset's usage complies with all licenses, laws, regulations, and ethical guidelines. In the case of any violation, we make no representations or warranties and accept no responsibility.



        


§ BROADER IMPACT


Our annotation campaign for the  dataset involved 20 human annotators providing over 52,000 annotations. This dataset offers valuable insights for advancements in machine learning research and raises important considerations in a broader context. 

1c, d Leveraging cost-efficient yet potentially error-prone annotators is vital in times of big data. However, requesting the support of human annotators, particularly through crowdsourcing, poses societal risks. On the one hand, human annotators can introduce noise into the dataset due to various reasons, e.g., missing expertise or lack of concentration <cit.>. Although our dataset is designed to mirror such human annotation noise, the reliance on such annotations requires careful reflection of their impact on generalization performance. On the other hand, annotation platforms often engage vulnerable individuals under challenging working conditions with low compensation <cit.>. It is essential to ensure fair compensation and working conditions.  Despite its potential benefits, collecting metadata about annotators poses privacy risks. Including only relevant information and rigorously protecting the annotators' privacy is essential to prevent misuse and data breaches <cit.>. 

To mitigate the risks inherent to human annotators' work, we implemented the following policies:


  * 5c Each annotator was employed as a student assistant at the University of Kassel and thus received a fair compensation following university standards. Specifically, student assistants without a first university degree received 12.41 per hour, and those with a first university degree 13.33 per hour. In both cases, the regulations of the German minimum wage were met.

  * 4d, e We implemented data privacy by following the General Data Protection Regulation (GDPR) <cit.>, which gave the annotators the right to reject the annotation work at any time and ensures that no annotator can be identified after the publication of the dataset. For this purpose, each annotator received a GDPR-compliant document to explain all annotators' rights and the purpose of the annotation campaign. After voluntary consent, the personal data of the annotators was only stored on the uni-internal storage media for the organization of the annotation campaign. The collected annotator metadata does not contain this personal data; it only includes task-related data. In addition, all personal data that would have revealed a mapping between annotators and their true identities was deleted from the uni-internal storage media before the dataset's publication.

  * 5a Healthy working conditions were ensured on the one hand by German labor laws and on the other by additional freedoms granted as part of the annotation campaign. Considering these labor laws, the annotators were allowed to work remotely and freely organize their working hours. This implies, in particular, that breaks could be taken at any time. Moreover, the annotators had the right to withdraw from the annotation campaign.

These policies ensured ethical standards and integrity throughout our annotation campaign.
        


§ RELATED DATASETS: ADDENDUM



In the main paper, we focused on related datasets that provide ground truth class labels alongside noisy annotations from multiple humans, are publicly available, and are frequently used in the literature for evaluating multi-annotator learning approaches. However, further datasets have similarities to . The  dataset <cit.> also involves annotating easily confusable animals. However, there are several key differences: Ground truth class labels for reliable evaluation are unknown. The dataset only provides annotations in the form of hard class labels with an estimated accuracy rate of about 92%, which is rather high compared to . Instead ofimages of animal species with groups of highly similar appearance, pairs of similar animals are distinguished. Finally, to gain access to , it is required to complete a form and submit it to the authors. <cit.> published a new collection of previously existing image datasets by adding annotations from multiple annotators. This collection is a benchmark to investigate the trade-off between the benefit of multiple annotations per instance and associated annotation costs (no annotation times). In contrast to , the datasets only contain hard class labels, and probabilistic class labels are obtained by combining class labels from multiple annotators. Two further datasets with class labels from multiple annotators are  and  <cit.>, where five annotators classified texts of software defect reports. Corresponding ground truth labels are not unknown. None of these datasets provide annotator metadata. Still, some datasets may make valuable contributions to evaluating multi-annotator learning approaches; therefore, we plan to implement these as part of our codebase. 
        


§ EMPIRICAL EVALUATION: ADDENDUM


This section details the main paper's experiments and associated results. More concretely, we discuss computational resources, present the hyperparameter search, and introduce annotator profiles.






 §.§ Computational Resources

3d We primarily performed experiments on the Slurm cluster of the IES group at the University of Kassel. This cluster is equipped with multiple NVIDIA Tesla V100 GPU and NVIDIA Tesla A100 GPU servers for machine learning and, in particular, deep learning research. However, we needed such a GPU server only once to cache the images of our dataset  as self-supervised features learned by the DINOv2 ViT-S/14 <cit.>. Subsequent benchmark and use case experiments were performed on CPU servers equipped with an AMD EPYC 7742 CPU. Additionally, we conducted small-scale experiments, e.g., creating the t-SNE <cit.> plot in the main paper, on a workstation equipped with an NVIDIA RTX 4090 GPU and an AMD Ryzen 9 7950X CPU. Using the CPU of this workstation, we measured the training times per epoch of the different approaches for the seven data set variants of . Fig <ref> presents the corresponding results as a bar plot. We observe that all approaches have training times in a similar range, where the two baselines are the fastest and  the slowest. For an upper estimate of the computation time of all experiments, we use 3s as training time per epoch to account for any overhead. For 50 training epochs, 45 · 10 hyperparameter experiments, 7 · 10 · 10 benchmark experiments, and 7 · 2 · 10 use case experiments (ignoring the fast experiments on annotation times in active learning), we obtain a total computation time of about 54h.





 §.§ Hyperparameter Search

3b We specified learning rate, batch size, and weight decay as hyperparameters for the RAdam optimizer <cit.> in a grid search, where the validation accuracy of the upper baseline  was our objective.  Table <ref> presents the corresponding results. This procedure ensures that hyperparameters are found with which the classification task can be learned. In addition, none of the approaches is disadvantaged or favored. Furthermore, an individual hyperparameter study per approach is unrealistic in practical scenarios due to the lack of a validation set with ground truth class labels <cit.>.





 §.§ Annotator Profiles

In the main paper, we presented the confusion matrix of the top-label predictions, the reliability diagram of the human-estimated likelihoods, and the histogram of annotation times across all annotators. Since the annotation behavior of the annotators is quite different, we now present these three aspects individually for each annotator. Together with selected metadata, we thus obtain a kind of profile per annotator in Figs. <ref>, <ref>, <ref>, and <ref>.

The analysis of the annotation times reveals that most annotators process the majority of their annotation tasks in rather short time frames (5s to 10s). However, a few annotators, e.g.,  (cf. Fig. <ref>) and  (cf. Fig. <ref>), take considerably more time to annotate an image. In combination with annotation times as a critical cost factor, active learning strategies <cit.> could aim to prefer fast-working annotators.

The variation between the individual reliability diagrams <cit.> of the annotators is large. On the one hand, there are some annotators for which higher top-label confidences imply higher accuracies, e.g.,  and  (cf. Fig. <ref>). On the other hand, there are also annotators whose top-label confidences are very unreliable, e.g.,  (cf. Fig. <ref>) and  (cf. Fig. <ref>). Quantifying the reliabilities and biases of such human-estimated likelihoods poses challenging research problems for the future.

We observe differences in error patterns by comparing the annotators' individual confusion matrices. For example, the annotator  (cf. Fig. <ref>) is an expert on distinguishing insects but struggles with correctly annotating squirrels. In contrast, the annotator  (cf. Fig. <ref>) is more accurate regarding squirrels while confusing insects more frequently. Such observations underline the importance of model annotator performance as at least a class-dependent variable.













    


 Paper Title: Few-shot Defect Image Generation based on Consistency Modeling.





 










Q. Shi et al.




Institute of Automation, Chinese Academy of Sciences School of Artificial Intelligence, University of Chinese Academy of Sciences CASI Vision Technology CO., LTD., Luoyang, China

{shiqingfeng2022, weijing2020, fei.shen, zhengtao.zhang}@ia.ac.cn

Few-shot Defect Image Generation based on Consistency Modeling
    Qingfeng Shi1,2 0009-0000-8480-9314Jing Wei1,2 0000-0001-7697-8070Fei Shen1,2,3()0000-0001-9263-4489 Zhengtao Zhang1,2,30000-0003-1659-7879
    August 5, 2024
===============================================================================================================================================

§ ABSTRACT 
Image generation can solve insufficient labeled data issues in defect detection. Most defect generation methods are only trained on a single product without considering the consistencies among multiple products, leading to poor quality and diversity of generated results. To address these issues, we propose DefectDiffu, a novel text-guided diffusion method to model both intra-product background consistency and inter-product defect consistency across multiple products and modulate the consistency perturbation directions to control product type and defect strength, achieving diversified defect image generation. Firstly, we leverage a text encoder to separately provide consistency prompts for background, defect, and fusion parts of the disentangled integrated architecture, thereby disentangling defects and normal backgrounds. Secondly, we propose the double-free strategy to generate defect images through two-stage perturbation of consistency direction, thereby controlling product type and defect strength by adjusting the perturbation scale. Besides, DefectDiffu can generate defect mask annotations utilizing cross-attention maps from the defect part. Finally, to improve the generation quality of small defects and masks, we propose the adaptive attention-enhance loss to increase the attention to defects. Experimental results demonstrate that DefectDiffu surpasses state-of-the-art methods in terms of generation quality and diversity, thus effectively improving downstream defection performance. Moreover, defect perturbation directions can be transferred among various products to achieve zero-shot defect generation, which is highly beneficial for addressing insufficient data issues. The code are available at https://github.com/FFDD-diffusion/DefectDiffu.




§ INTRODUCTION


Detection algorithms based on deep learning are crucial to industrial applications such as manufacturing<cit.>, transportation<cit.>, and power systems <cit.>. Efficient defect detection methods facilitate the identification of anomalies from normal products<cit.>. However, the performance of detection algorithms is often positively correlated with the number of annotated samples. With the improvement of production processes, defects become rarer in production lines and occupy only a small portion of the object<cit.>, posing challenges in data collection and annotation. Recently, image synthesis utilizing Generative Adversarial Networks (GANs)<cit.> and Denoising Diffusion Probabilistic Model (DDPM)<cit.> has attracted a surge of research<cit.>, leading to impressive progress in generating photo-realistic images<cit.><cit.><cit.>. Recent works<cit.><cit.><cit.><cit.><cit.><cit.> have demonstrated the feasibility of generating photo-realistic defect images to address the scarcity of defect images. However, existing methods face challenges when dealing with limited defect training samples, such as overfitting, poor diversity, and inaccurate annotations. 

These challenges stem from the fact that current methods primarily focus on tailored designs for specific scenarios, neglecting to address two key consistencies in industrial defect images. Firstly, various defects within a single product all originate from normal regions. As depicted in the first row of <ref>, normal regions of defect images and normal images are consistent, highlighting intra-product background consistency. Secondly, as shown in the second row of <ref>, defects of the same category across various products exhibit morphological similarities, illustrating inter-product defect consistency. Leveraging these consistencies, defect image generation can be divided into two steps: normal feature generation and defect feature transformation. Notably, the generation direction for the normal background should remain consistent within the same product, while the defect generation direction should be uniform across similar defects of different products, with the direction scale reflecting the strength of the defect. Compared with modeling the defect distribution of a single type of product, modeling the two consistencies across multiple products enriches available training data for generation. Therefore, modeling the two consistencies is crucial for generating high-quality and diversified defect images with limited training samples.

Recently, the pre-trained vision-language model CLIP<cit.> has showcased remarkable zero-shot capability, displaying robust generalization. This prowess is largely attributed to that CLIP aligns text and images to effectively model the consistency of target objects. In downstream tasks, CLIP is often utilized to model consistency information among images. In anomaly detection, WinCLIP<cit.> and AnomalyCLIP<cit.> leverage the pre-trained CLIP to model consistency in anomalies, achieving zero-shot anomaly detection. In image generation, text-to-image generation based on diffusion models has shown promising results. Text prompts are encoded by the text encoder of CLIP to provide consistent conditions for generation and guide diffusion models to generate images conforming to the text prompts, enabling diverse generation. DreamBooth<cit.> binds a unique identifier with the specific generation subject to achieve subject-driven generation. Custom Diffusion<cit.> optimizes only a few parameters in the text-to-image conditioning mechanism by four images, enabling the model to learn new concepts and achieve combinations of multiple concepts.
 
Based on the aforementioned analysis, we propose a novel text-guided diffusion method DefectDiffu, to model the intra-product background consistency and inter-product defect consistency across multiple products, achieving controllable diversified defect image generation with few-shot training samples. Firstly, we introduce a disentangled integrated architecture, comprising three key parts: background, defect, and fusion. Then multiple single-object text conditions are inputted into each part through a text encoder to facilitate the disentanglement and integration of background and defect, achieving the modeling of the two consistencies. Additionally, we extract cross-attention maps from the defect part to derive accurate binary mask annotations. Secondly, we propose the double-free strategy to concretize the two consistency directions. Double-free allows for flexible control over product types and defect strengths by adjusting the consistency perturbation directions. Finally, addressing the issues of poor generation of small defects and inaccurate masks, we introduce the adaptive attention-enhanced loss to boost the attention on defect regions, improving the generation quality of small defects and masks. Furthermore, we explore the ability of DefectDiffu for zero-shot defect generation by transferring defect perturbation directions. The main contributions of this paper are as follows:

1) We propose DefectDiffu to model intra-product background consistency and inter-product defect consistency across multiple products using text-guided disentangled integrated architecture, achieving controllable defect image generation with limited samples and obtaining masks during the generation.

2) We propose the double-free strategy to concretize background and defect perturbation directions, allowing for adjustment of perturbation scales to control product types and defect strength, and further exploring zero-shot defect generation capabilities.

3) We introduce the adaptive attention-enhanced loss to improve the generation quality of small defects and the accuracy of masks.


§ RELATED WORK



 §.§ Controllable Image Generation



  §.§.§ Diffusion-based generation.

Diffusion-based generation. In recent years, the diffusion models have experienced rapid development in image generation due to their high generation quality and training stability, surpassing GANs and VAEs (Variational Autoencoders)<cit.> in terms of the realism and diversity of generated images, becoming the most prominent image generation model, particularly in controllable image generation. <cit.> focuses on pre-training a classifier on noisy images, using the classification gradients of noisy images to guide category-controllable generation. <cit.> proposes classifier-free guidance, simultaneously training both conditional and unconditional models, using their differences as a substitute for an explicit classifier. DiT<cit.> replaces the commonly used UNet architecture in diffusion models with transformers to learn parameters of adaptive normalization layers from conditions, achieving state-of-the-art performance on the ImageNet.


  §.§.§ Text-guided Image Generation.

Recently, large-scale models based on diffusion models, such as DALLE 2<cit.>, Imagen<cit.>, and Stable diffusion<cit.>, have demonstrated impressive text-guided image generation capabilities, enabling precise control over the generated semantic contents<cit.><cit.>. Cross-attention mechanisms are the most commonly used method for introducing textual guidance into generation models, where cross-attention maps reflect positions corresponding to textual tokens in generated images. To ensure the alignment between generated images and text prompts, Attend-and-Excite<cit.>, MaskDiffusion<cit.>, and StructureDiffusion<cit.> modify cross-attention to enhance the focus of models on each token. Some works achieve label generation and image editing by processing cross-attention maps. DiffuMask<cit.> extracts cross-attention maps and obtains pseudo-masks for generated images through affinity networks. Attn2Mask<cit.> uses post-processed cross-attention maps as pseudo-masks for training segmentation models. Prompt-to-Prompt dynamically changes cross-attention maps during the generation to enable flexible image editing. Text-guided models demonstrate the ability to learn high-level semantic information from images <cit.> <cit.> <cit.>, allowing for customization generation by fine-tuning pre-trained models with a few images and textual descriptions.


 §.§ Defect Image Generation

Defect image generation is widely used to address insufficient data issues in detection tasks. Most current methods employ GANs to generate defect images with binary masks  (e.g., DefectGAN<cit.>, SDGAN<cit.>, AnomalyGAN<cit.>). MDGAN<cit.> synthesizes defects based on masks constructed from Perlin noise and defect annotations on real normal samples. <cit.> and <cit.> encode images into feature space to establish transformation directions between normal and defect images, and adjust transformation factors to achieve strength controllable defect image generation.  DFMGAN<cit.> pre-trains StyleGAN v2<cit.> on normal samples and fine-tunes with a few defect samples to update the proposed mask and defect generation modules. Currently, some works have explored few-shot defect image generation based on diffusion models. To improve generation quality, Defect-Gen<cit.> trains two diffusion models, the large model to determine the generated target structure and the small model for detail generation. However, these methods require training separate models for each product. Anomalydiffusion<cit.> generates masks by text flipping, adopting normal samples and masks as inputs to generate defect images for multiple products. However, predefined masks and normal samples limit the diversity of defect shapes and normal backgrounds. Moreover, the aforementioned methods fail to model the relationships between similar defects of multiple products, resulting in poor generalization. In contrast, DefectDiffu models image consistencies across multiple products and obtains masks based on cross-attention maps, thereby integrating features from multiple products to generate diversified defect images and avoid restrictions from predefined masks.


§ METHOD

As illustrated in <ref>, DefectDiffu adopts pre-trained DiT as backbones, consisting of the Condition Module, Generation Module, and the Mask Generation Module. The condition module employs the CLIP Text Encoder to introduce consistent text prompts into corresponding parts of the Generation Module. The generation module adopts a disentangled integrated architecture to disentangle the generation of normal backgrounds and defects, and then integrate them to generate realistic images under the guidance of text prompts. The mask generation module integrates cross-attention maps of the Defect Blocks to obtain mask annotations corresponding to generated images. Furthermore, the Adaptive Loss Ratio is designed to adjust the loss weights of the defect regions, enhancing the attention of DefectDiffu to defects. Next, we propose a double-free strategy to concretize the modeled consistencies, thereby achieving controllable generation of the product type and defect strength by modulating denoising results.



 §.§ Preliminaries

Image generation based on DDPM consists of the forward noise addition process and the reverse denoising process. In the forward process, Gaussian noise is added into the input image x_0 for T times to obtain the pure noisy image x_T,

    q(x_1:T| x_0):=∏_t=1^T q(x_t| x_t-1),q(x_t| x_t-1):=N(x_t;√(1-β_t)x_t-1,β_t I)

where  β_t is the variance at timestep t. In the reverse denoising process, the noise prediction model  ϵ_θ recovers x_0 from the noisy image x_t, where θ represents the learnable model parameters. ϵ_θ  takes  x_t and the current time step t as inputs to predict the removed noise at this time step. Furthermore, when implementing controllable image generation with text, images, or other information as condition c, ϵ_θ  takes x_t, the time step t, and the condition c as inputs to predict the noise. The training objective of the model is:

    min_θL=𝔼_x_t ,ϵ[‖ϵ-ϵ_θ(x_t,t,c)‖_2^2]



 §.§ Text-guided Disentangled Integrated Architecture

To model the intra-product background consistency and inter-product defect consistency, it is necessary to provide conditions that include consistency information for DefectDiffu and design architecture to achieve disentangled generation guided by consistency information. 
Firstly, as shown in <ref>, we adopt product names and defect names as intra-product consistency and inter-product defect consistency text prompts respectively, employing the CLIP text encoder to construct consistent guidance. Previous text-to-image generation methods used A photo of {object 1} {object 2}  {object n} to control the generation of multiple objects, which is not suitable for our work. The reason is that the text encoder encodes multiple objects in one text prompt into a fused output. Although this fused output can guide the model to generate high-quality images, it is difficult to disentangle multiple objects in the generation and may cause catastrophic neglect <cit.>. However, disentangling defects and backgrounds is the key point in modeling two consistencies of multiple products. Therefore, we use two single-object prompts as conditions to model consistencies, achieving the disentangled generation of backgrounds and defects. In addition, compared to the UNet-based diffusion architecture, DiT has a scalable hierarchical stacking architecture based on transformers. It allows us to introduce different conditions into each part of the model to learn the disentanglement generation of multiple objects.  

Therefore, as shown in <ref>, combining the advantages of single-object guidance and the flexibility of DiT, we propose the Text-guided Disentangled Integrated Architecture (TDIA). TDIA divides DiT into three parts, background, defect, and fusion. Background text embedding for modeling background consistency is introduced into the background part, defect text embedding for modeling defect consistency is introduced into the defect part, and the fusion part adopts a defect-background fusion text embedding to generate realistic defect images. Designed text prompt templates for each part are as follows:

Consistency condition c_p for Background Block:


A photo of {product name}


Consistency condition c_d for Defect Block:


A photo of {defect name}


Fusion condition c_f for Fusion Block:


A photo of {defect name} {product name}




 §.§ Double-free

Although DefectDiffu can generate diversified defect image ϵ_θ(x_t,t,c_d,c_p)  conditioned on c_p and  c_d after modeling two consistency by TDIA, the strength of generated defects can not be controlled and is limited to the training set distribution, which affects the diversity of generation. We hypothesize that there should be a perturbation direction along which normal images gradually become defect images, and the perturbation scale reflects the defect strength. Therefore, we perceive controllable defect image generation as a two-stage perturbation process involving both background and defect, with the defect strength adjustable by varying the perturbation scale. We set F_p^t  and  F_d^t as perturbation directions concerning two consistent conditions c_p and c_d at time step t. Then controllable generation can be achieved by adjusting the perturbation scale w_d and w_p to update the denoised results

    ϵ_θ(x_t,t,c_d,c_p)=ϵ_θ(x_t,t,∅,∅)+w_pF_d^t+w_dF_p^t

In order to calculate the two perturbation directions, we propose the double-free strategy based on the classifier-free guidance, achieving background and defect strength controllable generation beyond the diversity of the training set. First, considering ϵ_θ(x_t,t,c_g,c_p)  as conditional generation of normal background, where c_g is good, and unconditional background generation is realized by setting c_p to null to get ϵ_θ(x_t,t,c_g,∅), and the difference  F_p^t = ϵ_θ(x_t,t,c_g,c_p)-ϵ_θ(x_t,t,c_g,∅) is the perturbation direction for intra-product consistency, realizing controllable background generation. On this basis, inter-product defect consistency guidance is regarded as conditional generation of defect ϵ_θ(x_t,t,c_d,∅), corresponding unconditional generation is ϵ_θ(x_t,t,c_g,∅), and the difference  F_d^t = ϵ_θ(x_t,t,c_d,∅)-ϵ_θ(x_t,t,c_g,∅) is perturbation direction for inter-product defect consistency. Then we set the perturbation direction scale w_d of defects relative to normal images to achieve the controllable generation of defect strength. Finally, the <ref> can be simplified as:

    ϵ_θ(x_t,t,c_d,c_p)    =ϵ_θ(x_t,t,c_g,∅)+w_p(ϵ_θ(x_t,t,c_g,c_p)-ϵ_θ(x_t,t,c_g,∅))  
        +w_d(ϵ_θ(x_t,t,c_d,∅)-ϵ_θ(x_t,t,c_g,∅))

where w_p and w_d control the representation of the background and the strength of the defect respectively. The algorithm of double-free is shown in <ref>.
  



 §.§ Mask Generation

When training segmentation models with generated images, obtaining pixel-level mask annotations manually is laborious and time-consuming. Therefore, as shown in <ref>, we employ cross-attention mechanisms into the defect blocks and combine them with the mask generation module to simultaneously generate corresponding masks while generating defect images. Cross-attention is applicable for fusing the embeddings of the visual and textual features and producing spatial attention maps for each textual token. The noisy feature  φ(x_t)∈ℝ^H× W× C is projected to query vector Q∈ℝ^H× W× d, where H, W, and C is the length, width, number of channels of the tensor respectively, and d is the dimension of Q. The text embedding p∈ℝ^l× C with length l is projected to get key vector K∈ℝ^l× d and value vector V∈ℝ^l× d. Then cross-attention is calculated as,

    CrossAttention=Softmax(QK^T/√(d))

Next, we concatenate the cross-attention maps of multiple defect blocks along the channel dimension, followed by the mask generation module to obtain mask features M_f, and then decode  M_f to obtain pixel-level masks M_pre  with values between [0, 255] to locate generated defect positions. The real mask is encoded to feather Mask_f and the mask prediction loss is:

    loss_m=‖ M_f-Mask_f‖_2^2

With increasing time steps, the high-level semantic structure of the image will gradually emerge. Therefore, for accurate defect localization, we compute the average mask M_pre=1/100Σ_t=0^99M_pre^t  obtained from the last 100 time steps during testing and use an iterative method to binarize M_pre to obtain the final mask.


 §.§ Adaptive Attention-enhanced Loss

During training, we found that DefectDiffu tends to overlook defect features and generate normal images easily. Additionally, the generated masks are inaccurate for cases involving small defects or multiple defects within a single image. The reason lies in the fact that compared to normal regions, the features of defect regions with small proportions are more difficult to capture. Moreover, as shown in <ref>, the training objective of the diffusion model is to minimize the global reconstruction loss, where the reconstruction loss of defects contributes less to the overall loss. Consequently, the model has insufficient attention to defect regions, leading to poor reconstruction effects and inaccurate annotations. 
To address these issues, we propose the adaptive attention-enhanced loss based on cross-attention maps to increase the loss weight of defect regions, thereby enhancing the attention of DefectDiffu on defects. Firstly, we calculate the average cross-attention Map from extracted cross-attention maps of the defect block:

    Map=1/M∑_i=0^M Softmax(Q_iK^T/√(d))

where M is the number of extracted cross-attention maps. The real Mask is scaled to mask with the same size of Map to extract the attention for defect regions and normal regions, and then calculate the sum of each to obtain the adaptive attention ratio: 

    Rat=∑_i,j(1-mask)_i,j· Map_i,j/∑_i,jmask_i,j· Map_i,j +α

where i and j are height and width of Map, α is a constant used to prevent division by zero. The lower the attention allocated to the defect regions, the higher the Rat. 
To prevent the imbalance between defect and background regions caused by excessively high weights on defect regions, we further modify the magnitude of the attention ratio Rat to get R,

    R(Rat)={[      8,  Rat≥ 8;    Rat, 2<Rat<8;      2,  Rat≤ 2 ].

then the improved defect region loss is:

    loss_d=R‖ mask·ϵ_θ(z_t,t,c_d,c_p)-mask·ϵ‖_2^2

Finally, the training objective of DefectDiffu is:

    min_θ loss=‖ϵ_θ(z_t,t,c_d,c_p)-ϵ‖_2^2 + loss_d+λ loss_m

 where λ is hyperparameter.



 §.§ Zero-shot Generation


Based on the TDIA and double-free strategy, DefectDiffu can transfer defect features across normal backgrounds of different products, enabling zero-shot generation of defects. Only normal samples of the target product T are used in training, during testing, the perturbation direction of defect c_d^s learned from source products is added into the generation of normal images of T, resulting in generating defect samples of T with defect c_d^s. Specifically, during testing, the model is simultaneously fed with product prompt c_T and defect prompt c_d^s. Then the  perturbation direction of c_d^s modeled on other products is computed to update the denoised results at time step t, 

    ϵ_θ(x_t,t,c_d^s,c_T)    =ϵ_θ(x_t,t,c_g,∅)+w_p(ϵ_θ(x_t,t,c_g,c_T)-ϵ_θ(x_t,t,c_g,∅))  
        +w_d(ϵ_θ(x_t,t,c_d^s,∅)-ϵ_θ(x_t,t,c_g,∅))

 
Hence, DefectDiffu can generate defect images of T without requiring corresponding real defect images for training, which is attributed to that we successfully model the perturbation direction of the inter-product defect consistency.  



§ EXPERIMENT



 §.§ Experiment Settings



  §.§.§ Dataset and Evaluation Metrics.

 We conduct experiments on the MVTecAD dataset to validate the effectiveness of DefectDiffu. We select half of the images from the MVTecAD-test as the training set for image generation and baseline in segmentation experiments, with the remaining images used as the test set to evaluate the effectiveness of generated images in defect detection. All images are resized to 512×512. FID (Fréchet Inception Distance) and LPIPS (Learned Perceptual Image Patch Similarity) are calculated for quality and diversity evaluation respectively. A lower FID indicates higher generation quality. A higher LPIPS indicates better diversity. For the downstream application, we calculated mIOU (mean Intersection over Union), F1 score, mAP (mean Average Precision), and AUROC (Area Under the Receiver Operating Characteristic Curve) to evaluate the effectiveness of generated image-mask pairs, higher values indicate better performance.


  §.§.§ Implementation.
We set the N, M, and K in <ref> to 10, 10, and 8 respectively. The parameters of the corresponding modules are initialized with pre-trained DiT in ImageNet with the resolution of 512× 512. We set λ =0.2 in <ref>, p_1= 0.2, p_2=0.2, α=0.001 in <ref>. We use an iterative thresholding method for the binarization of generated masks. See the appendix for more details. 


  §.§.§ Comparison Methods.
 We employ StyleGAN2, DFMGAN, and Defect-Gen as comparison methods. StyleGAN2 has shown promising results in natural image generation and is commonly used as a foundational framework in industrial image generation. DFMGAN focuses on few-shot defect generation based on StyleGAN2. Defect-Gen utilizes two diffusion models to generate defects and their corresponding masks. They are the current state-of-the-art methods in industrial image generation with annotated masks. It is worth noting that these models need to be trained separately for each product. See the appendix for implementation details of the comparison methods.




 §.§ Comparison of Generation Results

<ref> shows the generated results of different methods, see the appendix for more results. It can be seen that Defect-Gen fails to model realistic high-level semantic features and the generated defects do not match real defect characteristics. DFMGAN generates unrealistic defects and contains numerous artifacts. StyleGAN2 fails to converge on leather and exhibits artifacts in products like cable. In contrast, DefectDiffu achieves good convergence across all 15 products. The generated results conform to real features, without mode collapse or artifacts, and the masks are accurate, validating the effectiveness of the proposed multi-product consistency modeling for few-shot image generation. Additionally, <ref> demonstrates the control of defect strength utilizing w_d. As w_d increases, the area of generated defects expands, and the number of defect regions increases, indicating a gradual increase in the severity of product damage. 

<ref> shows that our method achieves the optimal generation quality and good diversity on the MVTec dataset. Although the generation quality of toothbrush is slightly inferior to StyleGAN2, DefectDiffu simultaneously generates defect masks, which is more beneficial for training segmentation methods. Despite DFMGAN and Defect-Gen achieving higher LPIPS on some products, this is due to the introduction of unrealistic artifacts and the loss of real semantic features. It can be seen in <ref> and <ref> that the generation quality of these products is poor, corresponding to high FID. Clearly, diversity without adherence to real features is meaningless. In contrast, DefectDiffu achieves optimal generation quality on most products while maintaining good diversity. 


 §.§ Comparison of Detection Results. 


To validate the effectiveness of the generated results in industrial detection, we employ the real training set to train a UNet as a baseline and generate synthetic samples twice the size of real images by those methods capable of generating masks to augment the real dataset. Segmentation results are shown in <ref>. Compared to other methods, DefectDiffu can improve the performance of all products and achieve optimal segmentation results on most products. More results are provided in the appendix. In summary, DefectDiffu can efficiently improve the performance of detection methods and have high application value.


 §.§ Zero-shot Generation


As outlined in <ref>, we validate the zero-shot generation capability of DefectDiffu, with results shown in <ref>. Although DefectDiffu only learns the features of normal images from the target product during training, the defect consistency perturbation directions, modeled from other products, effectively guide the generation of defects for the target product. This enables DefectDiffu to achieve zero-shot defect generation without the need for real defect images of the target product during training. See the appendix for more results.


 §.§ Ablation Experiments



  §.§.§ Fusion part
. We remove the fusion part in the generation module, and the results are shown in <ref> w/o fusion. Without the fusion part, the semantic coherence between generated defects and backgrounds is disrupted, leading to a reduction in generation quality and diversity.


  §.§.§ AAL (Adaptive Attention-enhanced Loss)
. We set R to 3 in <ref> to train DefectDiffu, with results shown in <ref> w/o AAL. It indicates that without AAL, both the quality and diversity of the generated results decrease, indicating that AAL contributes to preventing the loss of defect features and enhancing the quality of generated defect images.


  §.§.§ Multiple single-object prompts
. To validate the role of multiple single-object prompts in TDIA for improving the generated quality of defect images, we only use the text prompt from the fusion part as the condition for DefectDiffu, and the quality of generated results are shown in <ref> OTP. It can be seen that employing a single text prompt to control the generation of multiple objects adversely affects generation quality.


  §.§.§ Defect strength scale w_d.
 We set w_d=1 and generate the same number of defect images at multiple strength scales to validate the effect of multi-strength generation on diversity. The LPIPS results are shown in <ref> single. It can be observed that under a single scale, the diversity is reduced, but it still outperforms the comparison methods in most products. This indicates that DefectDiffu effectively extends the diversity of the dataset through the double-free strategy.


§ CONCLUSION

This paper proposes DefectDiffu to achieve controllable generation of defect image-mask pairs across multiple products with limited samples. DefectDiffu leverages a disentangled integrated architecture and multiple single-object text prompts to disentangle defects and normal backgrounds, which facilitates the modeling of intra-product background consistency and inter-product defect consistency, thereby enhancing the diversity of normal backgrounds and defects. Furthermore, the double-free strategy is proposed to concretize the perturbation directions of the modeled background consistency and defect consistency, enabling multi-attribute controllable generation. Experimental results demonstrate that DefectDiffu outperforms state-of-the-art methods in both generation quality and diversity, effectively enhancing detection performance. Moreover, DefectDiffu achieves zero-shot defect generation based on inter-product defect consistency, reducing the costs associated with data collection and annotation.




§ ACKNOWLEDGEMENTS

This work was supported by the National Key Research and Development Program of China (2022YFB3303800), the National Natural Science Foundation of China (U21A20482), and Beijing Municipal Natural Science Foundation, China, under Grant L243018.





splncs04


[pages=-]09803-supp.pdf


 Paper Title: DriveArena: A Closed-loop Generative Simulation Platform for Autonomous Driving.























-1.0ex
    < g r a p h i c s >
 : A Closed-loop Generative Simulation Platform for Autonomous Driving
    Xuemeng Yang^1, *   Licheng Wen^1, *    Yukai Ma^2, 1, *   Jianbiao Mei^2, 1, *   Xin Li^3,5, *, † 

 
 
  Tiantian Wei^1, 4, *   Wenjie Lei^2, †   Daocheng Fu^1   Pinlong Cai^1   Min Dou^1

 

  Botian Shi^1,   Liang He^5   Yong Liu^2,   Yu Qiao^1 




^1 Shanghai Artificial Intelligence Laboratory   ^2 Zhejiang University   ^3 Shanghai Jiao Tong University 




^4 Technical University of Munich   ^5 East China Normal University




Project Page: <https://pjlab-adg.github.io/DriveArena/>

    August 5, 2024
==================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================

 
 
 
* Equal contribution,    Corresponding author 

† Work performed during internships at Shanghai AI Laboratory



§ ABSTRACT 
This paper presented , the first high-fidelity closed-loop simulation system designed for driving agents navigating in real scenarios.  features a flexible, modular architecture, allowing for the seamless interchange of its core components: , a traffic simulator capable of generating realistic traffic flow on any worldwide street map, and , a high-fidelity conditional generative model with infinite autoregression. This powerful synergy empowers any driving agent capable of processing real-world images to navigate in 's simulated environment. The agent perceives its surroundings through images generated by  and output trajectories. These trajectories are fed into , achieving realistic interactions with other vehicles and producing a new scene layout. Finally, the latest scene layout is relayed back into , perpetuating the simulation cycle. This iterative process fosters closed-loop exploration within a highly realistic environment, providing a valuable platform for developing and evaluating driving agents across diverse and challenging scenarios.  signifies a substantial leap forward in leveraging generative image data for the driving simulation platform, opening insights for closed-loop autonomous driving.

Code will be available soon on GitHub: <https://github.com/PJLab-ADG/DriveArena>







§ INTRODUCTION



Autonomous driving (AD) algorithms have advanced rapidly in recent decades <cit.>, progressing from modular pipelines <cit.> to end-to-end models <cit.> and knowledge-driven methods <cit.>. Despite demonstrating outstanding performance across various benchmarks, significant challenges persist in evaluating these algorithms on replayed open-loop datasets, obscuring their real-world efficacy.
Public datasets <cit.>, while offering realistic driving data with authentic sensor inputs and traffic behavior, are inherently biased towards simple straight-ahead scenarios. In such cases, an agent can achieve seemingly good performance by merely maintaining its current state, complicating the assessment of actual driving capabilities in complex situations. 
Furthermore, the agent's current decision does not affect execution or subsequent decisions in the open-loop evaluation, which prevents it from reflecting cumulative errors in real-world driving scenarios.
Additionally, the static nature of recorded datasets, where other vehicles cannot react to the ego vehicle's behavior, further hinders the evaluation of AD algorithms in dynamic, real-world conditions.

As illustrated in Figure <ref>, we analyze existing AD methods and platforms, revealing that most of them are inadequate for a high-fidelity closed-loop simulation.
Ideally, as an aspect of embodied intelligence, agents should be evaluated in a closed-loop environment, where other agents react to the actions of the ego vehicle, and the ego vehicle receives changed sensor input accordingly. However, existing simulation environments either cannot simulate sensor inputs <cit.> or have a significant domain gap with the real world <cit.>, making it difficult to seamlessly integrate algorithms into the real world, thus posing a huge challenge for closed-loop evaluation. 
We believe that the simulator should not only closely reflect the visual and physical aspects of the real world, but also promote the continuous learning and evolution of the model within an exploratory closed-loop system for adapting to diverse complex driving scenarios. To achieve this goal, it is imperative to establish a high-fidelity simulator that complies with physical laws and supports interactive functionalities. 

Therefore, we present , a pioneering closed-loop simulator based on conditional generative models for training and testing driving agents. 
Specifically,  offers a flexible platform that can be integrated with any camera-input driving agent. It adopts a modular design and naturally supports iterative upgrades of each module.  consists of a  that manages traffic flow and a  based on auto-regressive generation.
 can generate realistic interactive traffic flow on any road network worldwide, while  is a high-fidelity conditional generative model with infinite autoregression. The driving agent should make corresponding driving actions based on the images generated by , and feed them back to  to update the status of vehicles in the environment. The new scene layout will be returned to  for a new round of simulation. This iterative process realizes the dynamic interaction between the driving agent and the simulation environment. The specific contributions are as follows:

 
    
  * High-fidelity Closed-loop Simulation: We propose the first high-fidelity closed-loop simulator for autonomous driving,  , which can provide realistic surround images and integrate seamlessly with existing vision-based driving agents. It can closely reflect the visual and physical properties of the real world, enabling agents to continuously learn and evolve in a closed-loop manner and adapt to various complex driving scenarios.
    
    
  * Controllability and Scalability: 
    Our  can dynamically control the movement of all vehicles in the scenarios and feed the road and vehicle layouts into , which utilizes a conditional diffusion framework to generate realistic images in a stable and controllable manner. Additionally,  supports simulation using road networks from any city worldwide, enabling the creation of diverse driving scenario images with varying styles. 
    
    
  * Modularized Design: The Driving Agent,  and  communicate via network interfaces, enabling a highly flexible and modular framework. This architecture allows each component to be replaced with different methods without requiring specific implementations. Functioning as an arena for these players,  facilitates comprehensive testing and improvement of both vision-based autonomous driving algorithms and driving scene generative models.






§  FRAMEWORK


As illustrated in Figure <ref>, the framework of our proposed  comprises two key components: a  functioning as the backend physical engine and a  serving as the real-world image renderer. Unlike conventional approaches,  does not rely on pre-built digital assets or reconstructed 3D road models. Instead, the  adapts to road networks of any city in OpenStreetMap (OSM) format <cit.>, which can be directly downloaded from the Internet. This flexibility enables closed-loop traffic simulations on diverse urban layouts.

The  receives ego trajectories output by the autonomous driving agent and manages the movement of all background vehicles. Unlike world model approaches <cit.> that rely on diffusion models for both image generation and vehicle movement prediction, our  utilizes explicit traffic flow generation algorithms <cit.>. This approach enables the generation of a wider range of uncommon and potentially unsafe traffic scenarios, while also facilitating real-time collision detection between vehicles.

 generates realistic camera images that precisely correspond to the 's output. It also allows for user-defined prompts to control various elements of the generated images, such as street view style, time of day, and weather conditions, enhancing the diversity of the generated scenes. Specifically, it employs a diffusion-based model that utilizes the current map and vehicle layouts as control conditions to produce surround-view images. These images serve as input for end-to-end driving agents. Given 's closed-loop architecture, the diffusion model is required to maintain both cross-view and temporal consistency in the generated images.

The generated multi-view images of the current frame are fed into the end-to-end autonomous driving agents, which can output the ego vehicle's movement. The planned ego trajectory is subsequently sent to  for the next simulation step. The simulation concludes when the ego vehicle either successfully completes the entire route, crashes, or deviates from the road. Upon completion,  performs a comprehensive evaluation process to assess the driving agent's capabilities.

It is noteworthy that  employs a distributed modular design. The , , and AD agent communicate via network using standardized interfaces. Consequently,  does not mandate specific implementations for the  or the AD agent. Our framework aims to function as an “arena” for these “players”, facilitating comprehensive testing and improvement of both end-to-end autonomous driving algorithms and realistic driving scene generative models.



§ METHODOLOGY



Following the  framework outlined above, we have implemented a preliminary version of . In this section, we elaborate on the implementation of each module: , , and AD agent, while describing necessary details that were not previously mentioned. At the end of this section, we present both the open-loop and closed-loop evaluation metrics for AD agents in .



 §.§ 

Most existing realistic driving simulators <cit.> rely on limited layouts from public datasets, lacking diversity for dynamic environments. To address these challenges, we utilize LimSim <cit.> as the underlying  to simulate dynamic traffic scenarios and generate road and vehicle layouts for subsequent environment generation. LimSim also provides a user-friendly front-end GUI, which directly displays the BEV map and results from  and the driving agent.

Our  enables interactive simulations of multiple vehicles in traffic flow, including comprehensive vehicle planning and control. We adopt a hierarchical multi-vehicle decision-making and planning framework, which jointly makes decisions for all vehicles within the flow and reacts promptly to the dynamic environment through a high-frequency planning module <cit.>. The framework also incorporates a cooperation factor and trajectory weight set, introducing diversity to autonomous vehicles in traffic at both social and individual levels.

Furthermore, our dynamic simulator supports various custom HD maps of any city from OpenStreetMap, facilitating the construction of diverse road graphs for convenient simulation. 
The  controls the movement of all background vehicles. For the ego vehicle, we provide two distinct simulation modes: open-loop and closed-loop. In closed-loop mode, the driving agent performs planning for the ego vehicle, and  uses the agent-outputted trajectory to control the ego vehicle accordingly. In open-loop mode, the trajectory generated by the driving agent is not actually used to control the ego vehicle; instead,  maintains control in a closed-loop manner. The details of these two modes are further elaborated in Section <ref>.



 §.§ 



Unlike recent autonomous driving generation methods <cit.> that use Neural Radiance Fields (NeRF) and 3D Gaussian Splatting for environment reconstruction from logged video, we design a diffusion-based . It utilizes control conditions of the map and vehicle layouts from the  to generate geometrically and contextually accurate driving scenarios. Our framework shares several advantages: (1) Better controllability. The generated scenes can be controlled by scene layouts from , textual prompts, and reference images to capture different weather conditions, lighting, and scene styles. (2) Better scalability. Our framework can adapt to various road structures without the need to model the scene in advance. In theory, we support the generation of driving scenes for any city in the world by leveraging layouts from OpenStreetMap.

We illustrate our diffusion-based  in Figure <ref>. Built upon the stable diffusion pipeline <cit.>,  utilizes an effective condition encoding module that accepts a variety of conditional inputs including map and vehicle layouts, text descriptions, camera parameters, ego poses, and reference images to generate realistic surround-view images. Considering the importance of ensuring synthesis scene consistency across different views and time spans for driving agents, we integrate a cross-view attention module, inspired by <cit.>, to maintain coherence across different views. Additionally, we adopt an image auto-regressive generation paradigm to enforce temporal consistency. This approach enables  to not only maximally maintain the temporal consistency of the generated videos, but also generate videos of arbitrary length in an infinite stream, which provides great support for autonomous driving simulation.

Condition encoding. Previous work <cit.> applied BEV layout as conditional input to control the output of the diffusion model, which increased the difficulty of the network in learning to generate geometrically and contextually accurate driving scenes. In this work, we present a new condition encoding module to introduce more guidance information, which helps the diffusion module generate high-fidelity surround images. Specifically, in addition to encoding camera poses for each view, text descriptions, 3D object bounding boxes, and BEV map layouts using a condition encoder similar to <cit.>, we also explicitly project the map and object layouts onto each camera view to generate layout canvases for more accurate lane and vehicle generation guidance. 
Specifically, the text embedding e_text is obtained by encoding the text descriptions with the CLIP text encoder <cit.>. The parameters 𝐏 = {𝐊∈ℝ^3 × 3, 𝐑∈ℝ^3 × 3, 𝐓∈ℝ^3 × 1} of each camera and the 8 vertices of the 3D bounding boxes are encoded to e_cam and e_box by Fourier embedding <cit.>, where 𝐊, 𝐑, 𝐓 represent camera intrinsic, rotations and translations respectively. The 2D BEV map grid uses the same encoding method as in  <cit.> to get embedding e_map. Then, each category of the HD maps and the 3D boxes is projected onto the image plane to obtain the map canvas and box canvas, respectively. These canvases are concatenated to create the layout canvas. The final feature e_layout can be obtained by encoding the layout canvas by the conditional encoding network <cit.>.

Moreover, we introduce a reference condition to provide appearance and temporal consistency guidance. During training, we randomly extract a frame from the past 𝐋 frames as a reference frame and use the pre-trained CLIP model <cit.> to extract reference features e_ref from the multi-view images. The encoded reference features imply semantic context and are integrated into the conditional encoder through the cross-attention module. In order to make the diffusion model aware of the motion changes of the ego-vehicle, we also encode the ego-pose relative to the reference frame into the conditional encoder to capture the motion change trend of the background. The relative pose embedding e_rel is encoded by Fourier embedding.
By incorporating the above control conditions, we can effectively control the generation of surround images.

Auto-regressive generation. To facilitate online inference and streaming video generation while maintaining temporal coherence, we have developed an auto-regressive generation pipeline. Specifically, during the inference phase, the previously generated images and the corresponding relative ego pose are used as reference conditions. This approach guides the diffusion model to generate current surround images with enhanced consistency, ensuring a smoother transition and coherence with the previously generated frames.

What we designed in this paper is just a simple implementation of the . We also verify that extending the auto-regressive generation to a multi-frame version (using multiple past frames as references and outputting multi-frame images) and adding additional temporal modules can improve temporal consistency.



 §.§ Driving Agent

Recent works <cit.> have demonstrated the challenges in justifying the planning behavior of driving agents through open-loop evaluation on public datasets <cit.>, primarily due to the simplistic nature of driving scenarios presented. While some studies <cit.> have conducted closed-loop evaluations using simulators like CARLA <cit.>, discrepancies such as appearance and scene diversity persist between these simulations and the dynamic real world. To bridge this gap, our  provides a realistic simulation platform with the corresponding interfaces for camera-based driving agents <cit.> to perform more comprehensive evaluations, including both open-loop and closed-loop testing. Moreover, by changing the input conditions, such as the road and vehicle layouts,  could generate corner cases and facilitate these driving agents' evaluation on out-of-distribution scenarios. Without loss of generality, we select a representative end-to-end driving agent, namely UniAD <cit.>, to conduct both open-loop and closed-loop testing in our . UniAD utilizes surround images to predict motion trajectories for the ego vehicle and other agent vehicles, which can be seamlessly integrated with the API of our dynamic simulator for evaluation. Furthermore, the perceptual outputs, such as 3D detection and map segmentation, contribute to enhancing the validation of realism in our environment generation.





 §.§ Ego Control Modes and Evaluation Metrics



 inherently supports “closed-loop” simulation mode of driving agents. That is, the system adopts the trajectory output by the agent at each timestep, updates the ego vehicle's state based on this trajectory, and simulates the actions of background vehicles. Subsequently, it generates multi-view images for the next timestep, thus maintaining a continuous feedback closed-loop.
Additionally, recognizing that some AD agents may be unable to perform long-term closed-loop simulation during the development process,  also supports the “open-loop” simulation mode. In this mode, the  will take over the control of the ego vehicle, while the trajectory output by the AD agent is recorded for subsequent evaluation.

In both open-loop and closed-loop modes, it is crucial to comprehensively evaluate AD agent performance from a results-oriented perspective. Drawing inspiration from NAVSIM <cit.> and the CARLA Autonomous Driving Leaderboard <cit.>,  adopts two evaluation metrics: PDM Score (PDMS) and Arena Driving Score (ADS).


PDMS, initially proposed by NAVSIM <cit.>, evaluates the trajectory output at each timestep. We adhere to the original definition of PDMS, which aggregates the following sub-scores:

    PDMS_t=   (∏_m ∈{NC, DAC} score _m)_penalties×
       (∑_w ∈{EP, TTC, C}weight_w  × score _w/∑_w ∈{EP, TTC, C}weight_w)_weighted average .

where the penalties include the drive with no collisions (NC) with road users and drivable area compliance (DAC), as well as the weighted average including ego progress (EP), time-to-collision (TTC), and comfort (C). 
We implement minor modifications tailored to : in score _NC, we do not differentiate “at-fault" collisions, and for score _EP, we utilize the 's Ego path planner as the reference trajectory instead of the Predictive Driver Model.
At the end of the simulation, the final PDM Score is averaged across all simulation frames.

    PDMS = Σ_t=0^TPDMS_t/T∈ [0,1]


For open-loop simulations, PDMS serves directly as the evaluation metric for AD agents. However, for driving agents operating under the “closed-loop" simulation mode, we employ a more comprehensive metric called Arena Driving Score (ADS), which combines the trajectory PDMS with route completion:

    ADS = R_c×PDMS

where R_c∈ [0,1] represents route completion, defined as the percentage of the route distance completed by an agent. Given that “closed-loop" simulations terminate upon agent collision with other road users or deviation from the road, ADS provides a suitable metric for differentiating agents' driving safety and consistency.




§ EXPERIMENTS

 



 §.§  Setups

Dataset. 
For , we use the nuScenes <cit.> dataset for training. Following the official configuration, we employ 700 scenes for training and 150 for validation. We focus on four road categories (lane boundary, lane divider, pedestrian crossing, and drivable area) and ten object categories. The nuScenes dataset contains data collected from four different cities, covering various light and weather conditions, including daytime, night, sunny, cloudy, and rainy scenarios, enabling  to conditionally imitate diverse appearances. We additionally annotated each scene using GPT-4V, providing detailed scene descriptions that include elements like time, weather, street style, road structure, and appearance. These descriptions serve as text prompt conditions.






Model Setup. 
The model is initialized with the pre-trained Stable Diffusion v1.5 <cit.>, with only the newly added parameters being trained. For various conditions, except for the encoding of reference images and text prompts, the encoders for other conditions are randomly initialized and trained from scratch. These conditions are then integrated into the UNet using a randomly initialized ControlNet <cit.> to control the denoising process.

Training and Inference. 
To utilize the reference images and achieve temporal correlation, we employ ASAP <cit.> to generate 12Hz interpolated annotations and crop them into image clips of length 𝐋=7. During training, we use the last frame of each clip as the current frame, select any frame from the clip as the reference frame, and calculate the relative pose between them to model the motion trend of the background. Accordingly, the surround images corresponding to the reference frame are input to the network as reference images. During inference, the generated result of the previous frame is used as the current reference images, enabling unlimited length generation. 
The experiment is conducted on 8 NVIDIA A100 (80GB) GPUs with a batch size of 4×8 and 200K iterations of training. The AdamW optimizer is used with a learning rate of 1e-4.
The network follows the same image resolution (224×400) as MagicDrive, and when input to the driving agent, it will be upsampled to the original image size of nuScenes (900×1600) through a super-resolution algorithm <cit.>.



 §.§  Setups


Operating Frequencies.
In our experiments, the  operates at a frequency of 10Hz, while the control frequency is set to 2Hz. This configuration results in the  sending the current layout to  every 0.5 simulation seconds, requesting surround images. These images are then forwarded to the driving agent, which predicts and plans the subsequent trajectory for the ego vehicle. The , , and driving agent communicate via HTTP protocol, enabling deployment across different servers.

Simulation Modes.
As detailed in Section <ref>, we implement two simulation modes. In the open-loop mode, all vehicles, including the ego vehicle, are controlled by  itself. The driving agent can predict the ego vehicle's trajectory, but its trajectory is not actually executed. In the closed-loop mode, the ego vehicle is controlled by the driving agent, and the simulation terminates if it crashes with other vehicles or leaves the road.


Supported Maps.
Currently,  supports four different maps, which are: , , , and . The first two maps closely resemble the corresponding areas in the nuScenes dataset, while the last one replicates the road network of the Town05 map in the CARLA simulator. Notably,  can download road network data for any area directly from OpenStreetMap and perform simulations, enabling  to simulate the road network of almost any city worldwide. OpenstreetMap also accepts customized operations, and users can draw the desired road network structure for simulation testing.




 §.§  Fidelity Validation

To assess the sim-to-real gap between our generated images and the original nuScenes images, we employ UniAD <cit.> as an evaluator. We generate videos for 150 scenes based on the original layout provided by the nuScenes validation set with 2Hz. For comparative analysis, we set MagicDrive as the baseline method and use its official codes and checkpoints for inference. Subsequently, UniAD is performed on these images to compute various metrics, including 3d object detection, BEV map segmentation, and planning. The results are summarized in Table <ref>. It shows that all our indicators are higher than the baseline method, and a few indicators even surpass the performance on the original nuScenes. Furthermore, it demonstrates our model's superior capability to accurately respond to control signals and strictly adhere to input conditions. These findings establish a solid foundation for using our generator as a reliable simulator.









 §.§ Visualization

Controllability.
In this section, we will comprehensively demonstrate the controllability of the model from various dimensions, including the control of lighting and weather, the fit of object boxes and maps, change of street style, and consistency over long periods of time.

We demonstrate the impact of the reference image on the generated image, as shown in Figure <ref>. We randomly select one frame of images from the nuScenes dataset as reference images and choose three scenes from OpenStreetMap and Carla. We perform inference on them with  respectively. It can be seen that the source and style of the road network are very different from the scope of the original nuScenes dataset. The pictures show that the generated vehicles and road networks conform closely to control conditions, demonstrating strong control capabilities. The style and weather of the generated pictures can also be consistent with the reference images. In other words, besides maintaining image generation continuity through reference images, we can also regulate image style accordingly.

Figure <ref> presents images generated using different text prompts and reference images on the same road network. Each set of images portrays the surrounding scenery at intervals of 8.5 seconds and 24 seconds respectively, with the layout projected on the image. The images clearly 
illustrate that the road structure and vehicles strictly adhere to the given control conditions while maintaining excellent consistency in the surround view. In addition, the four sets of images exhibit significant differences in weather and lighting and can maintain their own styles during the continuous iteration process.


Scalability.
The  can accept any map downloaded from OpenStreetMap and seamlessly connect to the Carla road network. Combined with Dreamer's excellent following capability, the entire framework demonstrates robust scalability. The specific results are shown in Figure <ref>. We used both MagicDrive and our  to generate realistic images on the same Carla road network, with the corresponding lane lines projected onto the images. The road style in Carla differs significantly from that of nuScenes. It is rare to encounter roads with such large curvature and such wide roads in nuScenes. Consequently, the performance of MagicDrive, which is based on the nuScenes BEV map, is slightly inferior in these conditions. As indicated by the yellow arrow, MagicDrive struggles to generate curved roads and fit wide roads accurately. , however, can produce reasonable pictures that follow the road structure.


We also demonstrate additional cases using data from the nuPlan dataset to validate the scalability. The nuPlan data originates from cities different from nuScenes and features varying camera numbers and parameters. We select 6 cameras with a similar layout to the nuScenes dataset, and nuPlan's camera parameters are employed to project object boxes and lane lines onto corresponding images as control conditions. As shown in Figure <ref>,  fully trained on nuScenes adeptly adheres to these conditions, generating coherent images when deployed in new cities and even with novel camera configurations.


















 §.§ Open-loop and Closed-loop Experiments


In this section, we adopt the prevailing end-to-end autonomous driving method UniAD <cit.> as the driving agent to test both the open-loop and closed-loop performance within the  framework. We utilized UniAD's open-source code and pre-trained weights without additional training. UniAD operates at 2Hz, outputting a trajectory of 6 path points over the next 3 seconds.  further interpolates this to a 10Hz trajectory.

Open-loop Evaluation.
We first assess UniAD's performance in 's open-loop mode.
UniAD is evaluated on three scenarios: 1) the original nuScenes image sequences; 2) -generated nuScenes image sequences, where the vehicles' trajectory remains identical to nuScenes ground truth, but surround images are replaced with -generated ones; and 3) 's own simulation sequences (i.e., 's open-loop mode).
Our evaluation metrics consist of the PDM Scores and its sub-scores, as detailed in Section <ref>. Additionally, we evaluate trajectories driven by human drivers in nuScenes as the human driver performance. Detailed results are presented in Table <ref>.

The results reveal that while UniAD performs optimally on the original nuScenes sequence with a PDMS metric of 0.91, the -generated sequence surprisingly achieves a PDMS of 0.902, representing a metric drop of less than 1%. We attribute this to both the high fidelity of our -generated images and UniAD's strong dependence on ego states, as corroborated by <cit.>.

In 's open-loop mode, Figure <ref> illustrates two sequences, demonstrating that UniAD's prediction of the road network and vehicle tracking are fundamentally accurate. However, in terms of metrics, UniAD's performance in such scenarios with unseen road and traffic flow is significantly degraded, with an average PDM Score of only 0.636. The output trajectories by UniAD exhibit a substantial increase in collision rates and instances of driving outside the drivable area.
The open-loop experimental results underscore the critical importance of closed-loop experiments and tests for autonomous driving methods.

Closed-loop Evaluation.
We further evaluated UniAD's performance in 's closed-loop mode. In this mode, the trajectory outputted by UniAD is directly used for ego vehicle control, and the evaluation metrics include PDM Score (PDMS), Route Completion (RC), and Arena Drive Score (ADS). Our closed-loop experiment was conducted on four pre-set paths, with two paths selected in Boston and two in Singapore. The simulation time to complete each trajectory was approximately 120 seconds. Detailed results are presented in Table <ref>.

The results indicate that the PDMS of UniAD-generated trajectories in closed-loop mode (0.667) is comparable to that of the open-loop mode. However, the Route Completions (RC) are consistently low, averaging only about 13.7% of the total route length. Specifically, UniAD performs better on straightaways but largely fails to navigate the first turning intersection in the route. Figure <ref> illustrates two failure cases where UniAD lacked sufficient trajectory correction. Despite a roughly correct prediction of the road structure, it ultimately mounted the central green belt or failed to complete a right turn successfully. The average Arena Driving Score for UniAD is 0.086.
It should be noted that these are preliminary results based on testing only 4 routes. We plan to expand the number of routes for a more comprehensive evaluation and explore the combined effect of 's timing consistency and the driver agent's performance on the final ADS.




§ RELATED WORKS





 §.§ Data Acquisition for Autonomous driving


The characteristics of the automated driving dataset can be categorized into two aspects: appearance fidelity and interactivity. First, in terms of appearance fidelity, NGSIM <cit.> and CitySim <cit.> provide only realistic driving trajectories that can provide safe and reliable driving planning guidance. On top of that, some datasets developed based on the Carla simulator, such as DriveLM-CARLA <cit.> and BenchDrive <cit.>, provide simulated sensor data. Taking it a step further, the Waymo <cit.> and nuScenes <cit.> datasets capture real-world sensor recordings and the driving behavior of human drivers. The datasets were produced in a complex process and with a limited amount of data. To add variety to the scenarios, MagicDrive <cit.> and DriveDreamer <cit.> provide editable scenario generation. So far, we have been able to obtain diverse and rich data for training. However, the above data can only be used for open-loop evaluation, i.e., current decisions do not affect future data distributions, which differs significantly from real driving. Works <cit.> that also have fidelity differences, improve the interactivity of the data, they usually use auto-regressive generation methods to realize the interaction, the generation process implies the model's understanding of the world, and usually can not be too much human intervention. Some simulators <cit.> make things more controllable by decoupling part of the mechanics of how the world works. Common examples include simulators <cit.> that provide realistic traffic flow, and simulators <cit.> that drive vehicles in game engines, and reconstructive simulations represented by <cit.> that provide the appearance of reality.




 §.§ Diffusion-based Generative Models

Recent advancements in generative models have seen diffusion models play a pivotal role in image and video generation <cit.>. Moreover, recent works have expanded the scope by integrating additional control signals beyond traditional text prompts <cit.>. For instance, ControlNet <cit.> incorporates a trainable version of the SD encoder for control signals, while studies such as Uni-ControlNet <cit.> and UniControl <cit.> have emphasized the fusion of multi-modal inputs into a unified control condition using input-level adapter structures. 
Our approach aims to study the generation of continuous and controllable sequence frames, thereby bridging the gap between simulation environments and reality, and establishing the required foundation for closed-loop learning of autonomous driving agents.




 §.§ Evolution of Autonomous Driving Generation
 

World Models <cit.> utilize diffusion models to generate future driving scenes based on historical information, these methods often lack the ability to control the scenarios through layout, are difficult to generate continuous and stable videos and lack the approximation of physical laws.
TrackDiffusion focused on generating videos based on 2D object layouts <cit.>.
BEVGen <cit.> pioneered the generation of synthetic multi-view images based on the BEV layout, laying the foundation for a controllable generation of autonomous driving scenarios. BEVControl <cit.> extended this approach by a height elevation process, enabling image generation aligned with surrounding projection layouts.
Further advancements includes MagicDrive <cit.>, DriveDreamer <cit.>, Panacea <cit.> and DrivingDiffusion <cit.>, which generate panoramic controllable videos through various 3D controls and encoding strategies.
However, their primary focus lies in augmenting training data to enhance algorithm performance, rather than serving as simulators for modeling dynamic environmental interactions.



 §.§ Simulator-Driven Scenario Generation

Autonomous vehicle development is significantly enhanced by driving simulators, which provide controlled environments for realistic simulation. Prominent research efforts have concentrated on generating virtual imagery and annotations, with some studies expanding to incorporate environmental variations and construct safety-critical scenarios for training based on real-world data logs. Nevertheless, these simulated images frequently fall short of achieving true realism, as evidenced by previous works <cit.>. 
While SimGen <cit.> made a breakthrough as the first work to generate diverse driving scenarios following conditions from a simulated environment, it mainly focused on the quality of the generated content with only front-view images, neglecting the exploration of closed-loop systems. Our research aims to bridge this gap by developing a system that can not only generate realistic scenarios but also allow agents to interact with them in a closed-loop manner.




 §.§ Closed-Loop Driving in Simulation

End-to-end vehicle control algorithms <cit.>, are typically trained and evaluated on open-loop datasets <cit.>. However, these algorithms lack the capability to generalize directly to simulators for closed-loop evaluation, which hinders the demonstration of their true performance potential. Recent studies have increasingly recognized the significance of closed-loop evaluation, as exemplified by <cit.>. Moreover, simulation environments offer a wealth of training data, a stark contrast to models trained on datasets that are constrained by data distribution <cit.>. A significant challenge arises due to the discrepancy between the simulated scene's appearance and real-world conditions, complicating the generalization of models trained on simulation data to actual scenarios. This creates a paradox: the desire to utilize simulation data for its diversity and editability, while also seeking data that closely mirrors reality. Our approach effectively addresses this issue by enhancing the realism of the simulator for certain closed-loop learning methods <cit.>.




§ CONCLUSIONS AND FUTURE WORKS


This paper introduces a novel closed-loop simulation platform named  for vision-based driving agents.  integrates a  that generates human-like traffic flow and a high-fidelity generative  with infinite generation. This combination allows realistic interaction and continuous feedback between the driving agent and the simulation environment. The system provides a valuable platform for developing and testing autonomous driving agents in a variety of scenarios, marking a substantial leap in driving simulation technology.


 is designed with a modular architecture, allowing for easy replacement of each module. This paper presents an initial implementation of . As the first high-fidelity closed-loop simulator, we still have a few limitations for future improvement:

1) Data Diversity: The current generative model is trained solely on the nuScenes dataset, which limits the diversity and emergence capabilities. We plan to expand training to include more varied datasets to enhance the model's robustness and versatility.

2) Temporal Consistency: While we can generate continuous videos with an autoregression strategy, maintaining motion trends and temporal consistency between frames remains challenging. Future work will explore multi-frame autoregressive networks and more scalable architectures <cit.> to address these issues.

3) Runtime Efficiency: Like many generative models,  requires significant runtime. Investigating faster sampling methods <cit.> and model quantization may alleviate these problems.

4) Expanded Agent Testing: We plan to incorporate a broader range of driving agents within , facilitating the continuous learning and evolution of knowledge-driven driving agents in the closed-loop environment <cit.>.

5) A Real Arena:  can not only evaluate the performance of different driving agents, but also act as a testing ground for AD generative models. By using the same driving agent as a referee, it can fairly assess the sim-to-real gap of different generative models. This approach even provides a more credible and convincing evaluation compared to traditional metrics like FID and FVD.

We recognize that practical application may still be a way off, but the potential and promise shown by this work are evident. We hope this research will advance closed-loop exploration in highly realistic environments and offer a valuable platform for developing and assessing driving agents across a range of challenging scenarios. We encourage the community to collaborate in advancing this field. The era of open loops is transitioning, and autonomous driving evaluation and learning are set to enter a new era of closed-loop systems.



§ ACKNOWLEDGMENTS

The research was supported by Shanghai Artificial Intelligence Laboratory, the National Key R&D Program of China (Grant No. 2022ZD0160104) and the Science and Technology Commission of Shanghai Municipality (Grant Nos. 22DZ1100102 and 23YF1462900).




ieeetr





 Paper Title: Are Bigger Encoders Always Better in Vision Large Models?.








Are Bigger Encoders Always Better in Vision Large Models?
    Bozhou Li^†, Hao Liang^†, Zimo Meng^†, Wentao Zhang

Peking University


    August 5, 2024
============================================================================

§ ABSTRACT 

In recent years, multimodal large language models (MLLMs) have shown strong potential in real-world applications. They are developing rapidly due to their remarkable ability to comprehend multimodal information and their inherent powerful cognitive and reasoning capabilities. Among MLLMs, vision language models (VLM) stand out for their ability to understand vision information. However, the scaling trend of VLMs under the current mainstream paradigm has not been extensively studied. Whether we can achieve better performance by training even larger models is still unclear. To address this issue, we conducted experiments on the pretraining stage of MLLMs. We conduct our experiment using different encoder sizes and large language model (LLM) sizes. Our findings indicate that merely increasing the size of encoders does not necessarily enhance the performance of VLMs. Moreover, we analyzed the effects of LLM backbone parameter size and data quality on the pretraining outcomes. Additionally, we explored the differences in scaling laws between LLMs and VLMs.



#1#2
  [#1]#2

1†Equal Contribution



§ INTRODUCTION

In recent years, the rapid development of large language models has revolutionized the field of natural language processing  <cit.>. These models, equipped with an enormous number of parameters, have demonstrated outstanding performance in areas such as translation  <cit.>, question answering  <cit.>, and text generation  <cit.>. Due to the powerful capabilities of LLMs, a natural idea is to combine them with multimodal research, leveraging the strong cognitive abilities of LLMs to process information from other modalities  <cit.>. 

Among MLLMs, VLMs achieve competitive performance in traditional multimodal tasks such as image classification <cit.>, image understanding <cit.>, and image captioning <cit.>. Moreover, their excellent language understanding capabilities enable strong performance in text-rich tasks, such as vision question-answering <cit.> and image-text retrieval <cit.>.

Currently, the dominant architecture for VLMs employs a large language model as the backbone. Mechanisms such as cross-attention <cit.> or linear projectors <cit.> are utilized to connect the vision encoder (e.g., ViT <cit.>) with the LLM backbone <cit.>. Larger visual encoders possess stronger encoding capabilities, allowing them to extract features more effectively. These features are then transformed by the projectors for the LLMs to understand. We refer to this model architecture as the connected vision paradigm.

Apart from VLMs, previous research on scaling laws <cit.> adopts an empirical approach to study the relationship between model performance, parameter size, and the amount of training data. Under the assumption that the scaling trend follows a power-law relationship, researchers fit power-law formulas by training models of different sizes with varying amounts of data. Additionally, scaling laws have been extended to various scenarios, including graph neural networks <cit.>, data mixing <cit.>, data pruning <cit.>, and fine-tuning LLMs <cit.>. Although <cit.> investigated scaling laws in the context of multimodal scenarios, they neglected the connected vision paradigm <cit.>, leading to the following challenges:

C1. Poor Effectiveness: Since no scaling laws have been conducted, it is uncertain whether this paradigm can scale up to achieve better performance, resulting in suboptimal model performance.

C2. Low Efficiency: Without scaling laws, we do not know how much data or how many parameters to use. Typically, all available data are used for training VLMs, resulting in a waste of data and computational resources.

To address these issues, we conduct scaling law experiments following <cit.>. We select the renowned LLaVA1.5 <cit.> model as the backbone of our study. To investigate the scaling laws, we utilize models with 7 billion and 13 billion parameters. For the dataset, we choose image-text pairs from CC12M <cit.> and Laion400M <cit.>, known for their high-quality pairs. We use data sizes from 1 million to 10 million image-text pairs for our scaling law experiments.





The core contributions of this paper are summarized as follows:

    
  * New Perspective To the best of our knowledge, we are the first to conduct scaling law studies for the connect vision paradigm. Additionally, we are the first to analyze the limitations of this VLM paradigm.

    
  * New Observation We conducted experiments on the pretraining phase of MLLMs using the connect vision paradigm. From the experimental results, we derived a crucial conclusion: simply utilizing a ViT trained using CLIP with more parameters and better performance does not enhance the performance of MLLMs. This suggests that exploring alternative methods is necessary to improve the performance of MLLMs. We also indicate that this issue is highly likely not caused by limitations in the scaling ability of ViT.








§ BACKGROUND AND RELATED WORK



 §.§ Multimodal Large Language Model


In recent years, benefiting from the development of model architectures based on transformers  <cit.> and the increase in computing resources and training data, LLMs  <cit.> have emerged. These models possess massive parameters, requiring extensive computational resources to train on large datasets for a prolonged period. The sheer scale of these models, often exceeding billions of parameters, has enabled them to capture subtle nuances of language and learn complex patterns from vast amounts of text data. LLMs have not only demonstrated outstanding performance in traditional NLP tasks but have also exhibited emergent capabilities such as reasoning, code generation, and solving mathematical problems  <cit.>.

With the fast-paced advancement of LLMs, a natural question arises: how can we enhance the capabilities of LLMs to enable them to understand information from other modalities? Some earlier MLLMs, such as BEiT-3, attempted to train a multimodal model from scratch using data from multiple modalities  <cit.>. However, there is now a growing preference for combining pre-trained LLMs and modality encoders using specific mechanisms to construct multimodal LLMs. The methods combining pre-trained LLMs and modality encoders can be classified into four categories  <cit.>. The first and second categories employ cross-modal attention mechanisms  <cit.> and custom layers  <cit.>, respectively, to integrate information from other modalities into the internal representation of the LLM. These two types of methods are commonly referred to as deep fusion methods. The third category uses different modules to align information from other modalities with text information  <cit.>. Researchers exploring the fourth category attempt to unify the encoding of different modalities and expand the vocabulary of LLMs  <cit.>. These two methods are commonly referred to as early fusion methods. Due to their ease of training, high computational efficiency, and ability to construct any-to-any MLLMs, early fusion methods, especially the third category, which is also the focus of this article, are gradually becoming mainstream.

In general, the fusion modules used in the third category of methods can be classified into three types. The first type is transformer-based abstractors. These methods allow for adjusting the number of visual tokens to balance model performance and efficiency  <cit.>. However, they might lose locality inductive bias and require longer training times and more training data. The second type involves using MLPs (Multi-Layer Perceptrons)  <cit.>. This approach is lightweight, easy to train, and capable of capturing local information. However, it can only generate fixed-length visual tokens. Furthermore, some researchers have explored the use of custom layers to combine the advantages of both approaches  <cit.>.

The training process of  MLLMs can be divided into two stages: multimodal pretraining and multimodal instruction fine-tuning. During the multimodal pretraining (MM PT) stage, the input and output projectors are trained to align different modalities by optimizing predefined objectives. Typically, X-text pairs are used in this stage, where X represents data from other modalities. In the multimodal instruction fine-tuning (MM IT) stage, the pre-trained multimodal LLM is fine-tuned using a dataset formatted in an instructional manner. This fine-tuning aims to enhance the model's generalization ability on unseen tasks. In this article, we conducted experiments on the MM PT stage.







 §.§ Scaling Law


In the process of deep learning development, there has been a tendency to train larger models to achieve better performance. A natural question arises: as the number of model parameters and the amount of training data increase, how does model performance change? This has led to research on scaling laws. Hestness et al.  <cit.> were the pioneers in using empirical methods to study the scaling law of deep neural networks. Kaplan et al.  <cit.> and Hoffmann et al.  <cit.> conducted extensive experiments on transformer-based autoregressive models at different scales. They investigated the relationship between model loss and the number of model parameters and training data.

In situations where the model architecture is known, we can approximate the training cost based on the number of model parameters and the amount of training data (e.g., C ≈ 6ND for the transformer-based model, where N represents the number of model parameters, D represents the amount of training data, and C represents the computational costs). By combining this estimation with scaling laws, we can determine the optimal allocation of training data and model parameters within limited computational resources  <cit.>. This approach is particularly useful for LLMs since training them requires a significant amount of computational resources. Even if we embrace the philosophy underlying LLAMA  <cit.>, which entails setting aside computational expenses during the training phase and maximizing data usage to augment model performance at inference time, we consequently face an additional consideration: using larger models inherently translates to increased computational costs during inference. Therefore, even under conditions of ample data availability, a critical question arises: is the performance boost from expanding model parameter counts justified by the resultant higher computational overhead at inference? Under this assumption, scaling laws are still necessary.

After the initial scaling law was proposed, researchers studied the scaling law in different scenarios, such as graph neural networks  <cit.>, data mixing  <cit.>, data pruning  <cit.>, and fine-tuning of large language models  <cit.>. Some of them made certain adjustments to the form of the formula.

When it comes to MLLMs, Aghajanyan et al.  <cit.> explored the relationship between the loss and the dataset size and the parameter size over seven modalities, including text, image, image-text, speech, speech-text, code, and molecules. They also examined the scaling law when models are pre-trained on two modalities simultaneously and modified the form of the formula. However, the training paradigm they used in their study differs from the connected vision paradigm, the mainstream methods preferred today. They trained LLMs and modality encoders from scratch, similar to the approach chosen by BEiT-3  <cit.>, to obtain their scaling law. This discrepancy has sparked our research focus. To the best of our knowledge, the scaling law of multimodal large models under the connected vision paradigm is still under-researched.











§ METHODS



 §.§ Dataset


We utilized the CC12M dataset  <cit.> in our experiments, a comprehensive and expansive multimodal collection designed to foster advancements in artificial intelligence, particularly focusing on the intersection of vision and language. From the CC12M dataset, we strategically extracted subsets of data ranging from 1 million to 10 million images, incrementing by 1 million each time, to serve as our training datasets. This approach was chosen to systematically investigate the impact of varying training data sizes on the model's performance and learning outcomes. In our experiments involving different variants and scales of the Vision Transformer (ViT) trained by CLIP <cit.>, we maintained the use of identical datasets for each training session. This decision was made to eliminate the influence of random variables that could arise from using different data splits. By controlling for the input data, we aimed to isolate and observe the effects of model architecture and size on the experimental results, providing a clearer understanding of how these factors influence performance.

To validate the generality of our findings, we also conducted a set of experiments using the LAION-400M dataset  <cit.>. Similarly, we constructed training datasets ranging from 1M to 10M by randomly sampling from the LAION-400M dataset.

For each image-text pair in the dataset, we randomly selected a text prompt from the LLaVA paper's text prompts  <cit.> and randomly arranged its order with the image. We used the caption of the image as the ground truth. For each size of the training dataset, we extracted 25,600 of them as validation sets.



 §.§ Model Choice

In our quest to elucidate how different sizes of the ViT influence the performance of multimodal large language models, we deemed it crucial to employ ViTs of various sizes. To ensure that extraneous factors such as disparities in training data distribution or variations in training recipes did not skew our findings, we meticulously opted for a consistent series of ViT models throughout our experiments.

Our choice fell upon the ViT models  <cit.> trained using CLIP on the extensive Laion2B  <cit.> dataset, renowned for its comprehensive coverage and quality, thereby providing a robust baseline for our investigations. The specific model sizes we selected are shown in Table <ref>.





For the LLM backbone, the Vicuna series  <cit.> was selected. This model series was cultivated by fine-tuning the LLaMA2  <cit.> foundational model on a diverse set of user-shared dialogues sourced from ShareGPT  <cit.>. To ensure the reliability of our conclusions, we conducted training on both the 7B model and the 13B model from the Vicuna series.


 §.§ Training Process

In our overarching approach, we adopted the training process from the MM PT stage of LLaVA v1.5  <cit.>. Specifically, a two-layer Multi-Layer Perceptron (MLP) served as the linear projector within our framework, utilizing the Gaussian Error Linear Units (GELU)  <cit.> as the activation function to enhance non-linearity. Throughout the training phase, we froze the parameters of the ViT and LLM backbone and only updated the Linear Projector. 

We conducted the MM PT stage of the model using next token prediction as the training task and cross-entropy as the loss function. During the training process, we only masked the textual data in the ground truth and did not mask the image tokens.

Across a spectrum of dataset sizes, ranging from smaller to larger volumes, and varying the scale of ViT parameters and the size of the Vicuna model parameters, we trained the linear projector from scratch with a cosine scheduler. All experiments were conducted on 8*A100/A800 NVIDIA GPU machines.


§ EXPERIMENTS RESULTS

Tables  <ref> and  <ref> exhibit the concluding evaluation loss values obtained when employing various dataset sizes and ViT model sizes and training datasets built from CC12M. To render these outcomes more comprehensible at a glance, we have also produced Figures <ref>  and  <ref>. These visual aids aim to depict the relationship between model size, dataset size, and training loss, making the trends and patterns easier to discern. Table <ref> and Figure <ref> show the results with datasets built from LAION-400M.

Drawing insights from our experimental findings, several noteworthy observations emerge:

    
  * Increasing Data Quantity Improves Model Performance
    
    It is observed that for smaller training datasets, augmenting the data quantity effectively reduces the evaluation loss. This aligns with the learning theory that more examples lead to better model performance.
    
  * Larger LLM Backbone Enhances Model Performance 
    
    Comparing the training outcomes on the Vicuna-7B and Vicuna-13B models reveals that larger models consistently present lower evaluation loss than their smaller versions. This implies that larger language models possess a heightened capability to understand and interpret vision tokens.
    
  * The Importance of High-Quality Data
    
    The evaluation loss obtained from training on the CC12M dataset is lower than the evaluation loss from training on the LAION-400M dataset. The ViT models used in the experiment were trained on the LAION-2B dataset. Intuitively, its distribution should be closer to LAION-400M, and training multimodal LLMs with LAION-400M would yield better results. However, experimental results show that using CC12M is more effective. This is likely due to the quality of the datasets. Compared to LAION-400M, the CC12M dataset has higher quality, with a higher degree of matching between image and text. This result emphasizes the importance of high-quality datasets when training MLLMs.
    
  * Larger LLM Backbone Requires Less Training Data 
    
    Notably, when leveraging the Vicuna-13B model, the evaluation loss plateaus around the 7M mark in training data size, signifying a diminishing return on additional data. Conversely, the Vicuna-7B model continues to benefit from an increased data volume, illustrating a more gradual learning curve. This indicates that smaller models might require more substantial data increments to observe subsequent improvements, while larger models can exhibit more pronounced progress with less data.
    
  * Improved ViT Performance Doesn't Guarantee Better Results
    
    A critical finding from our study reveals that merely amplifying the parameter scale of ViT does not necessarily translate into enhanced model performance. Although vision transformers exhibit a relatively lesser scaling capacity compared with text transformers  <cit.>, the parameter scales selected in our experiments are significantly removed from the upper limits of ViT's scaling capabilities.

The 632M ViT trained using CLIP achieves a zero-shot accuracy of 78.0% on the ImageNet-1K dataset  <cit.> while increasing the model parameter size to 1.01B and training on a larger amount of data results in a zero-shot accuracy leap to 78.4%  <cit.>. Moreover, through architectural advancements, researchers have demonstrated ViT models boasting accuracies above 90%, with parameter counts around 20B  <cit.>. This underscores that within our experimental parameters, ViT's scaling potential has not been saturated, with enhanced performance still achievable through augmented model dimensions.

However, within our specific scenario, increasing the ViT's size does not yield superior results and, in certain instances, may lead to performance degradation. When training with the CC12M dataset, the validation loss of multimodal LLMs constructed with the 1.01B CLIP model's ViT is lower than that of models using the 632M model only when the training data volume is large. However, when training with data from the LAION-400M dataset, the validation loss of models using the 1.01B model consistently remains lower than that of models using the 632M model. This suggests that additional factors are at play, hindering the alignment and effectiveness between ViT and LLM in MM PT stage, necessitating deeper investigation.






















§ DISCUSSION & CONCLUSION

Our experimental outcomes underscore a critical observation: within the LLaVA  <cit.> framework, the simplistic augmentation of ViT's parameter scale does not correspondingly elevate the performance of MLLMs. This finding necessitates a strategic shift in our approaches to multimodal model design and optimization.

    
  * Strengthen Data Alignment Across Modalities With Data-centric Approaches
        
The experimental insights suggest the importance of improving the quality of training data for the alignment of different modalities. In this pursuit, data-centric approaches appear to hold significant promise. By focusing on enhancing data quality, variety, and relevance, we can potentially achieve better alignment between visual and textual modalities with higher efficiency, potentially leading to superior performance in multimodal applications  <cit.>.
        
    
  * Investigate The Differences Between Various Alignment Methods
    
    Deep Dive into Underlying Performance Limitations: A comprehensive investigation into why ViT performance enhancements do not translate into superior multimodal model performance is warranted. This exploration should encompass the dynamics of data utilization, model architecture, and training methodologies. When comparing the training approaches of CLIP  <cit.> and LLaVA  <cit.> we can observe certain differences in alignment strategies. The popular CLIP model utilizes a decoder-only LLM as the text encoder, such as GPT-2  <cit.>. In CLIP, the CLS token of the image processed by ViT is aligned with the EOS token of the text processed by GPT-2. In contrast, the processing approach of LLaVA is more akin to mapping the image tokens processed by ViT to the embedding space of a LLM using MLP layers. This differs from aligning image tokens with text tokens processed by GPT-2, as done in CLIP. The impact of these alignment method differences on model performance is worth exploring.


    
  * Exploration of Multimodal Information Fusion
    
        
  * Architectural Innovations for Multimodal Integration: There is a need to explore novel architectures that facilitate more seamless integration of multimodal information  <cit.>. This could involve designing frameworks that better enable the exchange and processing of visual, textual, and other modal data, potentially leading to enhanced overall performance.
        
  * Expansion of Vocabulary: Considering the inclusion of techniques like AnyGPT <cit.> to extend the vocabulary can prove beneficial in enriching the model's understanding and processing capabilities across different modalities. This could potentially enable the model to capture and utilize a broader spectrum of information, thereby enhancing its performance.
    


The observed phenomenon of larger LLMs requiring smaller datasets for alignment with ViT suggests a closer alignment between the semantic spaces of vision and text. This insight not only enriches our understanding of multimodal model dynamics but also opens avenues for optimizing data requirements and training strategies for MLLMs. Future research should delve deeper into the mechanisms behind these observations, aiming to leverage the inherent semantic alignment capabilities of larger LLMs for enhanced performance and efficiency in multimodal applications.



§ LIMITATIONS AND FUTURE WORK

Despite our study encompassing an extensive series of one hundred and twenty experiments across various parameter combinations, the investigation into the scaling laws of MLLMs during the MM PT stage remains a vast and complex field. Our current exploration, while insightful, is not exhaustive in addressing the comprehensive nature of this topic.

Compared to the scaling law of LLMs  <cit.>, the scaling law of MLLMs exhibits more complex characteristics.

    
  * More Influencing Factors
    
    More factors influence the performance of MLLMs, particularly when using pre-trained modality encoders and an LLM backbone to build MLLMs, the paradigm that is currently widely used. The parameter size of modality encoders, the amount of data used for pretraining the modality encoders and LLM backbone, the quality of data during the MM PT stage, and the distribution discrepancy between the data used for MM PT and the data used for pretraining the modality encoders and LLM backbone can potentially influence model performance. These factors may also interact with each other in more complex ways. In addition, different modalities, fusion methods for integrating multimodal information, and architectures of modality encoders may exhibit different forms of scaling law.
    
  * Different Estimates of Computational Costs
    
    Under the current training framework, it is not appropriate to solely rely on C≈ 6ND  for estimating computational costs. Due to the utilization of the backpropagation algorithm to update the model parameters, even if we only update the parameters of the MLP layer, which is quite lightweight, we still need to compute gradients for a significant portion of the LLM parameters. In addition, the modality encoder only needs to process data from other modalities, while the LLM also needs to handle text tokens. Taking LLaVA as an example, the compute budget for the MM PT stage of LLaVA can be estimated as:
    
    C ≈ 2N_ViTD_img+6N_LLM(D_img+D_txt)

    where N_LLM and N_ViT represent the number of parameters of LLM and ViT, and D_img and D_txt represent the number of image tokens and text tokens, respectively. The MLP layer is overlooked due to its relatively small number of parameters. It is worth noting that the value of D_img is dependent on the architecture of ViT, while the ratio between D_txt and D_img is also dependent on the source of the image-text pairs. For other fusion methods and modality encoder architectures, this formula may have different forms. This introduces new challenges in making trade-offs between computational costs and model performance.


To firmly substantiate the conclusions drawn in this paper, a more extensive empirical foundation is required. This necessitates conducting additional experiments on a broader range of LLM families, encompassing diverse architectures and capabilities. Moreover, the exploration should be extended to incorporate a wider variety of datasets—both in terms of content and scale—to ensure a comprehensive understanding of how multimodal models behave under different conditions.

Beyond empirical evidence, the surprising outcomes observed in our experiments demand a theoretical underpinning that goes beyond mere data analysis. The unexpected findings, such as the non-intuitive scaling behaviors of multimodal models or the peculiar interactions between vision and language semantic spaces, call for a deeper level of understanding that can only be achieved through rigorous theoretical analysis.

The prohibitive cost associated with extensive experimental setups has constrained our investigation to a specific scenario: the performance of large models where both ViT and LLM components are frozen. This approach, while providing valuable insights into the behavior of multimodal models under resource constraints, deviates significantly from the methodologies employed by some mainstream multimodal models such as Qwen-vl  <cit.>.

Our current experimental endeavors have been confined to the LLaVA architecture, providing a focused exploration of scaling properties within this specific framework. However, the landscape of multimodal AI architectures extends far beyond LLaVA, encompassing a diverse array of designs that include transformer-based connector architectures  <cit.> and VAE-based vision encoder architectures  <cit.>. Each of these architectural paradigms embodies unique characteristics and potential scaling behaviors that require systematic investigation.

Our experiments only considered MLLMs that can understand different modalities but can only generate textual information. We did not take into account more powerful yet complex MLLMs that can generate content in other modalities. The study of the scaling behavior of such models is more complex and requires additional computational resources.



unsrtnat



 Paper Title: RainMamba: Enhanced Locality Learning with State Space Models for Video Deraining.
















































Corresponding author. leizhu@ust.hk
	 
		   ^1The Hong Kong University of Science and Technology (Guangzhou)
     ^2The Hong Kong University of Science and Technology
      ^3Hong Kong Metropolitan University
      
		 
   

















































































§ ABSTRACT 
The outdoor vision systems are frequently contaminated by rain streaks and raindrops, which significantly degenerate the performance of visual tasks and multimedia applications. The nature of videos exhibits redundant temporal cues for rain removal with higher stability. Traditional video deraining methods heavily rely on optical flow estimation and kernel-based manners, which have a limited receptive field. Yet, transformer architectures, while enabling long-term dependencies, bring about a significant increase in computational complexity. Recently, the linear-complexity operator of the state space models (SSMs) has contrarily facilitated efficient long-term temporal modeling, which is crucial for rain streaks and raindrops removal in videos. Unexpectedly, its uni-dimensional sequential process on videos destroys the local correlations across the spatio-temporal dimension by distancing adjacent pixels. To address this, we present an improved SSMs-based video deraining network (RainMamba) with a novel Hilbert scanning mechanism to better capture sequence-level local information. We also introduce a difference-guided dynamic contrastive locality learning strategy to enhance the patch-level self-similarity learning ability of the proposed network. Extensive experiments on four synthesized video deraining datasets and real-world rainy videos demonstrate the superiority of our network in the removal of rain streaks and raindrops.

Our code and results are available at this https://github.com/TonyHongtaoWu/RainMambalink.
  







RainMamba: Enhanced Locality Learning with State Space Models for Video Deraining
    Hongtao Wu^1, Yijun Yang^1, Huihui Xu^1, Weiming Wang^3, Jinni Zhou^1, Lei Zhu^1,2
    
======================================================================================








§ INTRODUCTION


Videos captured from outdoor systems, , surveillance cameras and mobile sensors in autonomous vehicles, are often corrupted by both rain streaks and raindrops. 
These degradations significantly damage the visual perceptual quality and tend to degenerate the performance of subsequent outdoor computer vision and multimedia computing tasks, e.g., object detection <cit.>, semantic segmentation <cit.> and autonomous driving <cit.>. 

Therefore, rain removal is an indispensable pre-processing step to enhance the robustness and stability of outdoor intelligent systems and multimedia applications.


The earliest video deraining methods <cit.> were designed based on handcraft priors and attempted to model the physical characteristics and photometric properties of the rain layer to solve the problem. 

However, their performance is severely restricted by the laborious priors and they encounter complex optimization challenges.

In recent years, deep learning-based methods, including convolutional neural networks (CNN) and recurrent neural networks (RNN), have demonstrated effectiveness in video deraining tasks.

The success of video deraining methods <cit.> hinges on 
some elaborated modules, such as optical flow estimation and deformable convolutions, to leveraging temporal corrections from neighboring frames.

Nevertheless, these CNN-based methods usually have limited spatial-temporal receptive fields compared to recent transformer-based architectures <cit.>.

On the other hand, while these transformer-based methods achieve global understanding on videos, they heavily rely on Multi-Head Self-attention (MSA) mechanism, resulting in quadratic complexity related to sequence length.
This poses significant efficiency challenges in handling long video sequences <cit.> and impedes its application in modeling long-term inter-frame correlation.








Recently, State Space Models (SSMs) <cit.>, deriving from control systems theory, have demonstrated their advantages in natural language processing (NLP) <cit.> and computer vision <cit.> by their linear complexity in handling long sequences.

By formalizing discrete state space equations recursively, Mamba can capture long-range dependencies <cit.>, thereby improving video reconstruction quality by establishing global connections among pixels.

Nonetheless, the conventional Mamba model <cit.>, tailored for 1D sequential data in NLP, encounters inherent bottlenecks in video restoration tasks due to local pixel forgetting.

As illustrated in Fig. <ref>(a), the Mamba model recursively processes frames flattened into 1D sequences. 
This approach unexpectedly destroys the causality between spatio-temporally adjacent pixels in the video sequences, leading to the absence of local information in sequence-level temporal modeling.



To address the aforementioned issues, we devise an improved SSMs-based framework dubbed RainMamba for rain removal in videos.
We present a novel Hilbert scan mechanism that leverages the inherent locality characteristic <cit.> of Hilbert curve <cit.> to enhance the locality learning of the vanilla Mamba. 

In particular, we convert the spatio-temporal pixels into a 1D sequence following the trajectory of the Hilbert curve, thus leveraging the Hilbert curve's locality preservation for fine-grained restoration.

However, flattening video data into one 1D sequence inevitably diminishes the inherent local spatial correlations, hindering the restoration of rain streaks and raindrops areas.

Based on the observation that the patches in a given frame exhibit similarity with the neighboring counterpart in the same frame and subsequent frames, we propose a difference-guided dynamic contrastive locality learning strategy to preserve patch-level semantics.
Specifically, we derive rain location from the difference map for anchor sampling, while selecting the spatio-temporally surrounding patches as positive samples and spatio-temporally distant patches with significant differences as negative samples.

Motivated by Curriculum Learning <cit.>, we also introduce a dynamic mechanism that facilitates the optimization by adjusting the sampling distance for positive and negative samples.


In summary, our contributions are as follows:


  * We propose the first framework to adapt state space models to video deraining tasks by the Coarse-to-Fine Mamba Block.


  * We equip SSMs with a novel Hilbert scanning mechanism, 
which achieves localized scanning across both temporal and spatial dimensions.

This approach substantially improves our models’ ability to explore sequence-level local information.


  * We introduce a difference-guided dynamic contrastive locality learning approach to enhance the patch-level self-similarity learning ability.


  * Experimental results on four synthesized video deraining datasets and real-world rainy videos demonstrate that our network prominently outperforms state-of-the-art methods.







§ RELATED WORK



 §.§ Video Deraining


In the past decades, deep learning-based methods <cit.> have achieved impressive results for rain streak and raindrop removal in images.

SPANet <cit.> introduced the four-directional Initialize Recurrent Neural Networks to obtain the contextual information for rain streaks removal.
CCN <cit.> employed neural architecture search to adaptively find an optimal architecture for jointly tackling rain streaks and raindrops removal. 

D-DAiAM <cit.> leveraged the output differences between multiple deraining stages consisting of dual attention-in-attention model for joint removal of rain streaks and raindrops.
Recently, Video-level deraining techniques <cit.> have been developed to leverage temporal correlation and information across sequential frames to improve deraining results. 

FCRVD <cit.> constructed a two-stage recurrent network incorporating dual flow constraints to enhance motion information for video rain streaks and accumulation flow removal.

SAVD <cit.> employed deformable convolution to align multi-frame features, effectively removing both rain streaks and rain accumulation in videos.

RDD-Net <cit.> introduced the rain streak motion prior to a recurrent video rain streaks removal network.

ESTINet <cit.> employed long-short term memory to effectively capture spatio-temporal features and temporal correlations between neighboring frames.

Considering the distinct physical properties of raindrops, VWR <cit.> developed a spatio-temporal attention mechanism for video raindrops removal.

As rain streaks and raindrops frequently co-occur in videos, Wu  <cit.> devised a video deraining network ViMP-Net that integrates optical flow and mask-guided intra-frame 
 and inter-frame transformers to sequentially remove rain streaks and raindrops. 

In contrast, we introduce SSMs to causally model temporal information, replacing the mechanisms with low efficiency based on optical flow, deformable kernel or self-attention.





 §.§ State Space Models


Recently, State Space Models (SSMs) <cit.> have demonstrated notable efficiency in utilizing state space transformations <cit.> to manage long-term dependencies within language sequences.

S4 <cit.> introduced a structured state-space sequence model to exploit long-range dependencies with the benefit of linear complexity.

Based on this, Mamba <cit.> integrates efficient hardware design and a selection mechanism employing parallel scan (S6), thereby surpassing Transformers in processing extensive natural language sequences.

Subsequently, S4ND <cit.> explores SSMs' continuous-signal modeling of multi-dimensional data like images and videos.

More recently, Vision Mamba <cit.> and Vmamba <cit.> pioneered generic vision tasks, introducing bi-directional scan and cross-scan mechanisms to tackle the directional sensitivity challenge in SSMs.

Thanks to the superiority in addressing complex vision challenges, SSMs have been integrated into many vision tasks including object detection <cit.>, image segmentation <cit.>, image restoration <cit.>,
video object segmentation <cit.> and
video understanding <cit.>.

To the best of our knowledge, SSMs have not yet been explored in the video deraining task. 

Unlike previous SSMs-based works, we introduce a Hilbert scanning approach to enhance its sequence-level locality learning in spatio-temporal modeling.







 §.§ Contrastive Learning

Contrastive learning, an effective self-supervised learning technique <cit.>, seeks to minimize the distance between anchors and positive samples while maximizing the distance from negative samples within the representation space.

Contrastive learning has been explored in some low-level vision tasks like 
image translation <cit.>, image super-resolution <cit.>, image dehazing <cit.> , unsupervised image deraining <cit.>, image-to-image translation <cit.>,
video desnowing <cit.> and video deraining <cit.> . 


DCD-GAN <cit.> utilized contrastive learning to leverage features from unpaired clean images for guiding rain removal in the latent.

ANLCL <cit.> formulated an unsupervised contrastive learning based on the self-similarity property within samples and the mutually exclusive property between the rain layer and image layer.

UVDEC <cit.> constructed a cross-modal contrastive learning to investigate the mutual exclusion and similarity of rain-background layers between pixel domain and event domain.

The pivotal aspect of contrastive learning depends on the selection of anchor, positive, and negative samples.

Unlike previous image-level methods that sampled based on input and predictions or randomly selected patches, we developed a dynamic temporal sampling method to explore spatio-temporal self-similarity between video frames.




§ METHODS







 §.§ Network Architecture


Figure <ref> displays the overview of the proposed RainMamba for video deraining task. 

Specifically, our video deraining network is composed of a feature encoder, a series of cascaded Coarse-to-Fine Mamba modules, and a feature decoder. 

Given a rainy video sequence {I_t∈ℝ^3× H × W| t ∈[0, T)}, we employ a universal backbone (i.e., ConvNeXt <cit.>) and a lightweight head as the encoder to extract the shallow feature {E_t∈ℝ^C×H / 4 × W / 4| t ∈[0, T)}, where H and W denote the height and width of the input frames, and C signifies the number of channels.

To accommodate various patterns of rain in frames, we utilize convolutional layers to downsample and upsample the extracted features, thereby learning background semantics in a multi-scale manner.

In such a way, the feature is successively fed into Coarse-to-Fine Mamba Module (CFM) to causally learn the global and local temporal correlation and alignment by two different submodules at different scales, , global mamba block (GMB, see <ref>) and local mamba block (LMB, see <ref>).

Subsequently, after multi-scale globality and locality learning from temporal feature, we add N_3 CFM modules for further feature refinement.

Finally, the enhanced feature by CFM is processed through a reconstruction decoder, which is composed of multiple 3D convolutions and upsample layers, to generate the recovered frames
{J_t∈ℝ^3× H × W| t ∈[0, T)}.

During training, we design and enforce the difference-guided dynamic contrastive regularization to promote the network's dynamic learning of patch-level local self-similarity.



 §.§ Preliminaries



  §.§.§ State Space Models

Inspired by continuous linear time-invariant systems, Structured State Space Models (S4) and Mamba represent a class of sequence models that map a one-dimensional sequence x(k) ∈ℝ→ y(k) ∈ℝ through an intermediate hidden state h(k) ∈ℝ^N× 1, where N is the hidden state size.

Mathematically, SSMs utilize the ordinary differential equation (ODE) below to transform the input data:

    h^'(k)     =𝐀 h(k)+𝐁 x(k), 
    
    y(k)     =𝐂 h(k),

where 𝐀∈ℝ^N × N represents the system's evolution parameter, and 𝐁∈ℝ^N × 1, 𝐂∈ℝ^1 × N are the projection parameters, respectively. 


For practical application in deep learning, the continuous system described by Eq. <ref> is transformed into its discrete counterparts, through a discretization process using the zero-order hold (ZOH) method. This involves a timescale parameter Δ∈ℝ> 0, converting continuous parameters (𝐀, 𝐁) into discrete parameters (𝐀, 𝐁), which can be defined as follows:


    [                𝐀=exp (Δ𝐀),; 𝐁=(Δ𝐀)^-1(exp (Δ𝐀)-𝐈) ·Δ𝐁.;                            ]

This results in the following discretized model formulation:

    h_k=𝐀 h_k-1+𝐁 x_k, 
    
    y_k=𝐂 h_k.


To enhance computational efficiency and scalability, the Eq. <ref> can be mathematically transformed into an equivalent CNN form, leveraging parallel computation via a global convolution operation:


    M =    (CB, CAB, …, CA^L-1B) , 
       y = x⊛M,

where M∈ℝ^L is a convolutional kernel of the SSM, L is the length of the input sequence x and ⊛ represents the convolution operation.

Unlike traditional SSMs that employ constant transition parameters (𝐀, 𝐁), S6 <cit.> establish an input-dependent mechanism for matrices B, C and Δ, which enables better perception of input context information and dynamic updates of these parameters.




  §.§.§ Hilbert Curve




Hilbert curve <cit.> is a space filling curve (SFC) <cit.>, extensively deployed in various fields, including database <cit.> and image compression <cit.>.

As illustrated in Figure <ref>, the Hilbert curve has the ability to connect all elements within a space, and is often utilized as a fractal function <cit.>.


The Hilbert curve's defining characteristic lies in its strong capability to preserve locality <cit.> when transforming from one-dimensional to multi-dimensional spaces, significantly improving feature clustering <cit.>. 

Moreover, <cit.> shown that the Hilbert curve achieves better clustering in three-dimensional spaces. 

Based on this, Hilbert scan can enhance the correlation of spatio-temporally adjacent features by promoting clustering of neighboring tokens on the sequence.
More formally, a SFC can be denoted as p: [0,1] → [0,1] × [0,1], which maps any point from one-dimensional interval [0,1] to a coordinate in two-dimensional unit square.

We also denote n as the curve order of the Hilbert curve, which, in our discrete case, approximates to height and width of each frame. For any two points u,v in [0,1], their space to linear ratio (SLR) is defined as:

    SLR=|σ(u) - σ(v)|^2 /|u - v|.

The dilation factor (DF) of a SFC is defined as the upper bound of the SLR. For the same two points in [0, 1], if an SFC has lower DF, their mappings will also be closer in the unit square, which accords with the locality preserving requirement in the scanning stage. 

As proved in <cit.>, the DF of Hilbert curve is 6, while the normal Zigzag (row-and-column-order) curve is 4^n-2^n+1 + 2, which diverges to ∞ as the curve order n increases. 

Therefore, as the image resolution increases, the Hilbert curve can better maintain the locality of mapping any two points on a one-dimensional sequence to a multi-dimensional space than the Zigzag curve.

Also, the DF can provide explicit mathematical interpretation for comparing the impact of two scans.

Specifically, compared to Hilbert scan's locality, Zigzag scan can capture more global correlations.

Our research extends the locality of Hilbert curves to video sequences for enhancing the preservation of local pixel information during spatial-temporal scanning of SSMs. In this manner, the two scanning mechanisms can complement each other in establishing both global and local correlations.





 §.§ Coarse-to-Fine Mamba Module


Previous works <cit.> tend to elaborate some modules based on optical flow, deformable kernel or quadratic-complexity self-attention, to exploit temporal information with low efficiency for rain removal in videos.

Our method introduces the Structured State Space Models (dubbed Mamba), which can employ the selective scan mechanism to causally process the temporal data with linear complexity.


Specifically, we flatten the 3D video data into one-dimensional sequences in two different ways to effectively leverage spatio-temporal corrections from neighboring frames.

Our proposed Coarse-to-Fine Mamba Module consisting of Global Mamba Block and Local Mamba Block can progressively mitigate the degradation through holistic and regional multi-scale perception.





  §.§.§ Global Mamba Block



As illustrated in Figure <ref>, in Global Mamba Block, we apply a zigzag order approach (i.e. row by row) to construct global flattening.

This global scanning mechanism enables SSMs to establish global contextual dependencies between each pixel and all other pixels in a linearized sequence.

We transform the input video feature E∈ℝ^C× T×H / 4 × W / 4 into a one-dimensional long sequence V∈ℝ^C × (T × H/ 4× W/ 4) in a spatially-prioritized manner. 
Then, the flattened sequence V is fed into a Mamba Block.

We incorporate the Bi-Mamba layer <cit.> into our Mamba block for video deraining.
This Bi-Mamba layer processes flattened visual sequences through a bidirectional fashion, , forward and backward SSMs, which has been proven to be effective on low-level video tasks <cit.>.

Subsequently, a depth-wise convolution layer with the kernel size of 3 × 3 × 3  is employed to preserve fine-grained details.

The operation within the stacked Mamba Block can be defined as follows:

    V^l=BM(LN(V^l-1))+V^l-1, V^l=DWC(LN(V^l))+V^l .

Finally, we reshape the output one-dimensional sequence features back to their original three-dimensional features Ê∈ℝ^C× T×H / 4 × W / 4 in the global flattened order.



  §.§.§ Local Mamba Block


While global scanning facilitates modeling temporal information by causally processing time series data with SSMs, it destroys the inherent local correlations in videos. 

As shown in Fig. <ref>(a) and (c), the global scanning approach significantly increases the distance between spatially and temporally adjacent pixels, which leads to severe adjacent pixel forgetting.

To overcome this challenge, we present a novel Hilbert scanning technique to enhance the locality learning of video data during scanning.

The proposed strategy is illustrated in Fig.<ref>(b) and (d), highlighting the contrast between our method and global scanning approach.

As shown in Fig. <ref>, we first construct a 3D Hilbert curve that traverses every point following the principles of Hilbert curves. 

In essence, the 3D Hilbert algorithm is designed to recursively subdivide the cubic space and generate Hilbert curve segments within each smaller cubic space. 
These smaller segments are then merged to construct the full 3D Hilbert curve.

Subsequently, we transform each coordinate (x̂ , ŷ , ẑ ) on the 3D Hilbert curve into a unique one-dimensional index, thereby flattening the three-dimensional space by the Hilbert curve's scanning trajectory. 

This indexing formula can be represented as:

    Index = x̂· Q · T + ŷ· T + ẑ,


where Q is the width length of input features. 
Based on the Hilbert index, we flatten the input features Ê∈ℝ^C× T×H / 4 × W / 4 into the one-dimensional sequence V̂∈ℝ^C × (H/ 4× W/ 4× T) which is locally enhanced against the global sequence V.

Afterward, the sequence is fed into the Mamba block, which has the same network architecture as the global counterpart, to exploit spatio-temporal corrections in the local spirit.

Finally, we reshape the output sequence back to its original shape to construct the locally-enhanced features Ê^'∈ℝ^C× T×H / 4 × W / 4.

In contrast to the global scanning approach, our Hilbert scanning method is designed to more effectively capture sequence-level local dependencies across both temporal and spatial dimensions. 

This benefits from the Hilbert curve's inherent capacity to preserve locality, as highlighted in Sec. <ref>.

By cascading Global Mamba Block and Local Mamba Block, our network first obtains the holistic understanding of spatio-temporal information and then preserves its local details following the spirit of Coarse-to-Fine.  











 §.§ Difference-Guided Dynamic Contrastive Locality Learning


Although Mamba can effectively take advantage of the causality of videos by flattening 1D causal sequential data for temporal modeling, 
it frequently underestimates inherent 2D spatial relationships. 

To address this, we introduce a regularization strategy to enhance the model's intra- and inter-frame perception of spatial correlations.

Inspired by non-local prior <cit.>, we encourage the locality learning of spatial-temporally neighboring pixels by constructing a contrast learning mechanism that exploits the self-similarity from the patch level.

The rain in videos exhibits varying intensities and types, making rain removal more challenging especially when rain streaks and raindrops exist simultaneously.

According to <cit.>, the t-th video frame containing rain streaks and raindrops can be represented as:

    R S D_t=(1-M_t) ⊙   (B_t+S_t)+M_t⊙ D_t,

where R S D_t denotes the rainy video frame containing the clean background layer B_t, rain streaks layer S_t and raindrops layer D_t.

M_t is a binary mask, and M_t(x)=1 means pixel x is a part of the region occluded by raindrops.

Based on Eq. <ref>, we can derive the position information of rain streaks and raindrops by the difference map Ω_t between rainy and clean frames.

    Ω_t=(1-M_t) ⊙ S_t + M_t⊙ D_t- M_t⊙ B_t.


As illustrated in Fig. <ref>,

it's observed that regions containing streaks and raindrops typically correspond to high-response areas on the difference map. 

We adopt the average threshold method to select patches with high response in the difference map from the restored frame {J_t| t ∈[0, T)} as anchors {O_t| t ∈[0, T)}. 
We focus the attention of recovery on heavily degraded areas by eliminating the patches with lower response than the average response.

Moreover, we can observe that patches of the target frame share similarities with their adjacent counterparts in the same frame and also subsequent frames.

Conversely, these patches are significantly different from the spatio-temporally distant patches.
Based on these observations, we select spatially adjacent patches from the neighboring clean frames {Ĵ_j| j ∈[t-1, t+1]} as positive samples {P_j| j ∈[t-1, t+1]}.

On the other hand, we treat spatially distant patches from the entire degraded input sequence as negative samples {N_w| w ∈ [0, T)} and enhance their distinctiveness from positive samples by performing data augmentation such as rotation, flipping, and blurring.

Furthermore, we introduce dynamic learning to facilitate the optimization of intra- and inter-frame locality learning.
We incrementally increase the spatial distance p between positive samples and the anchor, and simultaneously decrease the spatial distance d between negative samples and the anchor during the training process. The entire dynamic learning process can be represented as:



    d = max(d_0 ·θ ^e/m, d_min),



    p = min(p_0 + e/m· (p_max - p_0), p_max),

where d_0 is the initial minimum negative distance, θ denotes the decay rate, e is the number of completed training steps, m is the total number of training steps, p_0 indicates the initial positive range and p is the sampling distance of positive samples.
This dynamic learning approach enables the network to progressively master patch-level details, advancing from simple to complex concepts.


Ultimately, we employ contrastive learning to ensure the s-th sample O_t^s is pulled closer to positive samples P_j^s and pushed far away from the strongly augmented degraded negative samples N_w^s through the pre-trained VGG feature extractor. 

Our proposed contrastive learning loss can be formulated as: 

    ℒ_DCL = 1/S∑_s=1^S( ∑_r=1^2ℒ_L_1(G_r(P_j^s), G_r(O_t^s))/ℒ_L_1(G_r(N_w^s), G_r(O_t^s))) ,

where {G_r| r ∈[1, 2]} extracts the r-th low-level hidden layer features from the pre-trained VGG-19 <cit.> model.  




 §.§ Loss Function

We adopt the Charbonnier loss <cit.> and the perceptual loss <cit.> to improve the visual quality of the restored results. 

The perceptual loss is to quantify the discrepancy between the features of the prediction and the ground truth. 
The overall supervised loss is formulated as:


    ℒ_sup =ℒ_pixel+λ_1ℒ_perceptual +λ_2ℒ_DCL,

where λ_1 and λ_2  are the balancing hyper-parameters, empirically set as 0.3 and 0.1, respectively. And the perceptual loss is formulated as follows:

    ℒ_perceptual=ℒ_M S E(V G G_3,8,15(Ĵ_t), V G G_3,8,15( J_t)) .






§ EXPERIMENTS














































 §.§ Dataset and Metric

In this section, we conduct a comparative evaluation of our video deraining network against state-of-the-art methods on four benchmark datasets, including two video rain streak removal datasets, a video raindrop removal dataset, and a video rain streak and raindrop removal dataset. We utilize the peak signal-to-noise ratio (PSNR), the structural similarity index (SSIM) <cit.>, and the learned perceptual image patch similarity (LPIPS) <cit.> to quantitatively compare different methods. 

Video Rain Streak Removal Datasets. 
These two video rain streak removal datasets are RainVID&SS (Rain Video Detection and Semantic Segmentation)<cit.> and RainSynAll100 <cit.>. 

RainVID&SS includes 205 short clips from ImageNet-VID and 3 long clips from CamVid for training. The training set has 86 short clips and 2 long clips. 

The RainSynAll100 dataset comprises 900 videos for training and 100 videos for testing.


Video Raindrop Removal Dataset. 

LWDDS (Large-scale Waterdrop Dataset for Driving Scenes) <cit.> is the first synthetic video waterdrop dataset for raindrop removal in driving scenes. 

This dataset has 67,500 triplets from 45 videos for training, and 600 triplets obtained from 6 videos for testing.


Video Rain Streak and Raindrop Removal Dataset. 

VRDS <cit.> is the only video raindrop and rain streak removal dataset with a total of 102 videos.
72 videos with 7200 frames are used for training, while 30 videos with 3000 frames are for testing. 
 





 §.§ Implementation Details.


Our network is trained on NVIDIA RTX 4090 GPUs and implemented on the Pytorch platform.

Note that we have four benchmark datsets for testing our network and compared methods.
Due to the page limit, we here provide the training details of the VRDS dataset. Specifically, at each training iteration, the input frame is randomly cropped to a spatial resolution of 256×256, and the number of frames per video clip is 5. 

The total number of the training iteration is 300K. 

We adopt the Adam optimizer <cit.> and the polynomial scheduler with a power of 1.0. 

The initial learning rate of our network is set to 5× 10^-4 with a batch size of 8 and a warm-up start of 2k iterations. 
Moreover, the encoder adopts the ImageNet pre-trained ConvNeXt <cit.> backbone.

The number of Coarse to Fine Mamba Modules N_1, N_2, and N_3 are set to 2, 3, and 2 respectively.

For training details on the other three datasets, please refer to Supplementary Material. 



 




 §.§ Comparisons with State-of-the-art Methods



  §.§.§ Quantitative Comparisons on the VRDS Dataset.


As shown in Tab. <ref>, we quantitatively compare our proposed method with 12 state-of-the-art (SOTA) image and video deraining methods, including CCN <cit.>, PreNet <cit.>, DRSformer <cit.>, MPRNet <cit.>, Restormer <cit.>, S2VD <cit.>, SLDNet <cit.>, ESTINet <cit.>, RDD <cit.>, RVRT <cit.>, BasicVSR++ <cit.> and ViMP-Net <cit.>.

Among the 12 compared methods, ViMP-Net has the best PSNR, SSIM, and LPIPS performance, and the PSNR, SSIM, and LPIPS scores are 31.02 dB, 0.9283, and 0.0862.
Moreover, our method has better metric results than ViMP-Net, and our PSNR, SSIM, and LPIPS scores are 32.04 dB, 0.9366, and  0.0684.

It indicates that the state space model with enhanced locality learning enables our framework to achieve a better temporal modeling ability, when compared to the optical flow, deformable kernel, and self-attention of SOTA compared methods.




  §.§.§ Quantitative Comparison on RainVID&SS Dataset.


Tab. <ref> reports the quantitative results of our proposed method and 6 SOTA video deraining methods on the two testing sets, which are ImageNet-VID+ dataset and the Cam-Vid+ dataset.
These 6 SOTA video deraining methods are MSCSC <cit.>, FastDrain <cit.>, PReNet<cit.>, SLDNet<cit.>, S2VD <cit.>, and MPEVNet <cit.>.

Regarding the ImageNet-VID+ dataset, our method achieves the largest
PSNR score of 35.07 dB and the largest SSIM score of 0.9561.
As the first place, our method outperforms the second place (i.e., MPEVNet) by a PSNR margin of 1.24 dB and a SSIM margin of 0.0109.

On the Cam-Vid+ dataset, our video deraining network achieves the largest PSNR score of 32.65 dB and the largest SSIM score of 0.9328, which outperforms all compared six methods. 

It indicates that our method can more effectively remove rain streaks and has a better capability in significantly improving the image quality of input rainy videos with rain streaks.

 



  §.§.§ Quantitative Comparison on RainSynAll100 Dataset.

Tab. <ref> reports the quantitative results of our network with 6 SOTA video deraining methods, and they are  FastDerain <cit.>, FCRVD <cit.>, RMFD <cit.>, BasicVSR++<cit.>, NCFL <cit.>,
and SALN <cit.>. 

SALN has the better PSNR and SSIM scores among the 6 SOTA methods.
Compared to SALN, our method improves the PSNR score from 29.78 dB to 32.16 dB, and enhances the SSIM score from 0.9315 to 0.9446.
Our superior PSNR and SSIM performance experimental results demonstrate our proposed technique significantly surpasses all competing methods in terms of removing rain streaks from rainy videos.




  §.§.§ Quantitative Comparison on LWDDS Dataset.


Tab. <ref> summarizes the PSNR and SSIM scores of our proposed method and 6 SOTA video raindrops removal methods on the LWDDS datasets.
These 6 SOTA video raindrops removal methods are CCN <cit.>, Vid2Vid <cit.>, VWR<cit.>, BasicVSR++<cit.>, ViMP-Net <cit.>, and SALN <cit.>.

From these quantitative results in Tab. <ref>, we can find that SALN has the largest PSNR score of 36.57 dB, and the largest SSIM score of 0.9802 among all compared 6 SOTA video raindrops removal methods.

More importantly, our method further improves the PSNR score from 36.57 dB to 37.21 dB, and the SSIM score from 0.9802 to 0.9816.
It also verifies the superior effect of our method in terms of the video raindrop removal task. 




  §.§.§ Visual Comparisons

Fig. <ref> visually compares the results of removing rain streaks and raindrops from video frames on the VRDS dataset with different rainfall intensity and lighting conditions.
Our network demonstrates superior performance in recovering clean background images, particularly excelling in raindrop areas where it more effectively restores detailed texture.
These visual results demonstrate that our network excels in recovering image areas affected by raindrops and rain streaks. 
For instance, in the third sample set, our approach effectively maintains the coherence and completeness of the window area impacted by rain, while also retaining its natural coloration.

Our network demonstrates superior preservation of non-rain background details by leveraging state space models for causally temporal modeling, combined with enhanced locality learning at both the sequence and patch levels.

Moreover, Figure  <ref> shows the visual results of our network and state-of-the-art methods on real-world rainy video frames.

These visual results indicate that our method can more effectively remove rain streaks and raindrops in real-world driving scenes and during outdoor video recordings, making it more applicable to common outdoor multimedia applications.









  §.§.§ Model Complexity and Efficiency Comparison


As reported in Table <ref>, we compare the number of parameters, FLOPs, and running time of our network and state-of-the-art methods on a NVIDIA RTX 4090 GPU.

The GFLOPs and Runtime are calculated by inferring a video clip of five frames with a resolution of 256×256.

We follow  <cit.> to calculate the GFLOPs metric.

And the runtime indicates the time needed to process each frame during inference.

Thanks to the linear complexity of SSMs and the critical components of our network, our approach achieves significant improvements in both effectiveness and efficiency.

For more discussions on the model efficiency, please refer to Supplementary Material. 

















 §.§ Ablation Study



  §.§.§ Baseline Design.


We perform ablation experiments to verify the effectiveness of three critical components of our network, and they are the Global Mamba Block (GMB), the Local Mamba Block (LMB), and the Difference-Guided Dynamic Contrastive Locality Learning (DCL) of our RainMamba.

Initially, we establish a baseline (denoted as “M1”) by removing all three major components from our network.

Subsequently, the Global Mamba Block is incorporated into the baseline model “M1" to build “M2". 

Following this, “M3" is constructed by integrating the Local Mamba Block into “M1".

To build “M4", the Global Mamba Block is integrated into “M3", thereby resulting in a complete Coarse-to-Fine Mamba Module.

Finally, we add our contrastive learning strategy to “M4" to reach a full setting of our RainMamba model.








  §.§.§ Quantitative Comparison.

Tab. <ref> presents the quantitative results of our proposed method alongside the four baseline networks (i.e., “M1" through “M4") on the VRDS datasets. 

Specifically, compared with “M1”, “M2” improves the PSNR score from 29.35 dB to 30.86 dB, the SSIM score from 0.9118 to 0.9277, and the LPIPS score from 0.0893 to 0.0774. 

It demonstrates the effectiveness of our Global Mamba Block in employing state space models to capture long-range dependencies among sequential frames.

Also, “M3" demonstrates a further metric improvement, which indicates that the Local Mamba Block can improve the video restoration quality of our network by enhancing locality learning.

Furthermore, “M4" significantly advances beyond M3, which proves the effectiveness of incorporating local scanning and Hilbert scanning mechanism together to model temporal information in a sequence-level manner.

Moreover, our network further outperforms “M4", which indicates that leveraging our contrastive learning strategy benefits the locality learning of our network in a patch-level manner, thereby improving the video deraining performance of our network.




  §.§.§ Comparisons of Hilbert-based Scanning directions.

We alse compared three  main Hilbert-based scanning directions on model performance in Table <ref>. 

Starting from the same point, there are three  main scanning directions, namely width-first, height-first, and time-first.
The result indicates that our model is insensitive to the Hilbert-based scanning directions,

since Hilbert scan enhances the overall spatio-temporal locality between the adjacent tokens within one-dimensional sequence.





  §.§.§ Quantitative Comparison and Analysis of Long Video Processing on NTURain Dataset.

We also conducted comparisons of our model with other video deraining methods on NTURain dataset for video rain streak removal. From these quantitative results in Tab. <ref>, Our method demonstrates an enhancement in performance over the ESTINet <cit.>. 

We also explored the potential of our model in handling ultra long videos.

As reported in Table <ref>, we input full resolution video clips and compare the experimental results from using video clips of different lengths.

The experimental results indicate that as the input frame rate increases, the effectiveness of video restoration also improves.

This demonstrates that the long-sequence modeling capability of SSMs can effectively leverage the spatio-temporal contextual information in videos to successfully remove rain streaks.

For more visual results and discussions on NTURain dataset, please refer to Supplementary Material. 


































§ CONCLUSION

In this work, we present a novel video deraining framework RainMamba with the improved state space models. 
To the best of our knowledge, we are the first to apply state space models to achieve effective rain streaks and raindrops removal in videos.
To better adapt Mamba to video deraining tasks, we introduce a Hilbert scanning mechanism to preserve the regional details based on the sequence level, which is the core element of our Local Mamba Block.
Moreover, a difference-guided dynamic contrastive locality learning is designed to further enhance the local semantics from the patch level.
Experimental results on four video-deraining benchmarking datasets demonstrate the superiority of our proposed framework.
We believe this will be a compelling baseline for the future of state space models in the community of low-level vision tasks.









This work was supported by the Guangzhou-HKUST(GZ) Joint Funding Program (No. 2023A03J0671), This work is supported/funded by the Guangzhou Municipal Science and Technology Project (No. 2024A04J4230), the Guangzhou Industrial Information and Intelligent Key Laboratory Project (No. 2024A03J0628), the Nansha Key Area Science and Technology Project (No. 2023ZD003), and Guangzhou-HKUST(GZ) Joint Funding Program (No. 2024A03J0618).



















ACM-Reference-Format







[
\begin@twocolumnfalse





t]c
    RainMamba: Enhanced Locality Learning with State Space Models for Video Deraining

    

 
    – Supplementary Material – 

    
 


.5em


\end@twocolumnfalse
]


In this supplementary material, we present network complexity (Section <ref>), more ablation studies (Section <ref>) and extra visual demonstration (Section <ref>). In addition, a video demo is provided
to showcase the dynamic display of our proposed local scanning mechanism and the effectiveness of our method at this https://github.com/TonyHongtaoWu/RainMambalink.


§ MODEL ANALYSIS







 §.§ Model Complexity and Parameters Comparison


As reported in Table <ref>, we compare the number of parameters, FLOPs, and running time of our network and state-of-the-art methods on a NVIDIA RTX 4090 GPU.

The GFLOPs and Runtime are calculated by inferring a video clip of five 
frames with a resolution of 256×256.

We follow  <cit.> to calculate the GFLOPs metric.

And the runtime indicates the time needed to process each frame during inference.

As shown in Figure <ref>, we demonstrate the effectiveness of our RainMamba by achieving state-of-the-art results on the VRDS datasets while maintaining a comparatively minimal computational expense. 

In the presented tabular data, our method obtained the best restoration performance results compared to other comparative methods and achieved the fastest speed.

Our model boosts a 4.68 dB improvement in PSNR compared to the second fastest method ESTINet <cit.>. 

This relatively fast inference speed enables our model to be effectively utilized in real-world applications.

Thanks to the linear complexity of state space models and the critical components of our network, our approach achieves significant improvements in both performance and speed.













§ MORE ABLATION STUDIES










 §.§ Visual Results of Ablation Study
 

As shown in Figure <ref>, in addition to quantitatively comparing the ablation experiments of RainMamba on VRDS dataset, we also conducted visual comparisons to qualitatively verify the effectiveness of three critical components of our network.

By introducing the Global Mamba Block (GMB), our “M2” model effectively eliminates a significant number of artifacts associated with raindrops and rain streaks, compared to “M1”.
However, the “M3” model can not preserve the spatial structure of certain details effectively and introduced extensive artifacts outside the window.

Leveraging Local Mamba Block (LMB), our “M3” model achieves superior detail retention, such as the shape of windows.

The integration of GMB and LMB significantly enhances the detail recovery in areas obscured by raindrops, as our “M4” model improves the modeling ability of spatiotemporal information.

By combining these three complementary contributions, our RainMamba clearly removes the artifacts and better restores the scene structures.









 §.§ Visual Comparisons of Different Scanning Mechanisms
 
Figure <ref> illustrates the results of “M2” and “M3” model.

The “M2” model can generate incorrect directional extensions when reconstructing the shapes of objects obscured by raindrops. 

This issue arises because “M2” utilizes a global scanning approach (row-and-column-major order), which neglects spatio-temporal continuity and leads to local pixel forgetting.

Our proposed local scanning mechanism improves the network’s local information awareness by rearranging the scan path of Mamba in sequence-level temporal modeling.





§ MORE EXPERIMENTAL RESULTS





 §.§ More Implementation Details

The encoder extract multi-scale feature maps (i.e., scales of 1/4, 1/8, 1/16, 1/32), after which a lightweight head merges these maps to generate encoded features E_t. 

We set the hyperparameters for different datasets according to the original paper's settings.

For the RainVID&SS<cit.> and RainSynAll100<cit.> datasets, input frames are randomly cropped to a spatial resolution of 128×128, with the number of frames per video clip being 7 and 5 respectively. For the LWDDS dataset <cit.>, the input frame is cropped to 256×256, with 5 frames per clip. The initial learning rate for our network is set at 2× 10^-4 for RainVID&SS and RainSynAll100 datasets, and 4× 10^-4 for the LWDDS dataset. A consistent batch size of 4 is used across these three datasets.









 §.§ Quantitative and Qualitative Comparisons on the NTURain Dataset

We also conducted comparisons of our model with state-of-the-art video deraining methods on a widely-utilized NTURain dataset for video rain streak removal.

NTURain <cit.> dataset contains 25 videos for training and 8 videos for testing.

From these quantitative results in Tab. <ref>, Our method demonstrates an enhancement in performance over the ESTINet <cit.>, improving the PSNR score from 37.48 dB to 37.87 dB, and the SSIM score from 0.9700 to 0.9738. 

These results demonstrate the capability of our method to effectively remove rain streaks in videos without incorporating any additional physical priors.

Figure <ref> visually compares rain streak removal results predicted by our network and state-of-the-art method ESTINet <cit.> from the NTURain dataset

Compared with ESTINet, our network demonstrates superior performance in restoring the original background images by effectively eliminating rain streaks from input video frames.




 §.§ Analysis of Long Video Processing


We selected the NTURain dataset as our test dataset due to its lower frame resolution (640×480) and the extensive sequence length of its videos (ranging from 116 to 298 frames).

Our experiment is implemented on a NVIDIA RTX A6000 GPU with a graphics memory of 48 GB.

We initially select 7 frames for input according to the training setting, and subsequently increased the number of input frames in increments of 10.

As reported in Table <ref>, we input full resolution video clips and compare the experimental results from using video clips of different lengths.

The experimental results indicate that as the input frame rate increases, the effectiveness of video restoration also improves.

This demonstrates that the long-sequence modeling capability of SSMs can effectively leverage the spatio-temporal contextual information in videos to successfully remove rain streaks.

It is noteworthy that our RainMamba is capable of processing 110 frames of full-resolution video simultaneously on a single GPU.

Due to the critical role of inter-frame information in video restoration tasks, the long video processing capabilities of our RainMamba are anticipated to be applicable to other video restoration challenges.



 §.§ More Results on the Compared Datasets
 
Figure <ref>, Figure <ref> and Figure <ref> demonstrate more visual comparisons between the results generated by our methods and the other compared methods on the RainVID&SS dataset, the RainSynAll100 dataset and the LWDDS dataset, respectively.
We also presented the visual results and comparison of our method and the other compared method on the NTURain dataset's real-world dataset in Figure <ref>.

The results from three datasets show that our RainMamba effectively removes various rain patterns, including streaks and raindrops of different sizes.

Also, our approach preserves the most natural color compared to alternative comparative methods.










































 Paper Title: Detecting, Explaining, and Mitigating Memorization in Diffusion Models.




Coordinating Decisions via Quantum Telepathy
    [
    
============================================



§ ABSTRACT 
Recent breakthroughs in diffusion models have exhibited exceptional image-generation capabilities. However, studies show that some outputs are merely replications of training data. Such replications present potential legal challenges for model owners, especially when the generated content contains proprietary information. In this work, we introduce a straightforward yet effective method for detecting memorized prompts by inspecting the magnitude of text-conditional predictions. Our proposed method seamlessly integrates without disrupting sampling algorithms, and delivers high accuracy even at the first generation step, with a single generation per prompt. Building on our detection strategy, we unveil an explainable approach that shows the contribution of individual words or tokens to memorization. This offers an interactive medium for users to adjust their prompts. Moreover, we propose two strategies i.e., to mitigate memorization by leveraging the magnitude of text-conditional predictions, either through minimization during inference or filtering during training. These proposed strategies effectively counteract memorization while maintaining high-generation quality. Code is available at <https://github.com/YuxinWenRick/diffusion_memorization>.





§ INTRODUCTION

Recent advancements in diffusion models have revolutionized image generation, with modern text-to-image diffusion models, such as Stable Diffusion and Midjourney, demonstrating unprecedented capabilities in generating diverse, stylistically varied images. However, a growing body of research <cit.> reveals a concerning trend: some of these “novel” creations are, in fact, near-exact reproductions of images from their training datasets, as depicted in the top row of <ref>. Some of the creations appear to borrow elements from these training images, as illustrated in the second row of <ref>. This issue of unintended memorization poses a serious concern for model owners and users, especially when the training data contains sensitive or copyrighted material. A poignant real-life example of this is Midjourney, which felt obliged to ban prompts with the substring “Afghan” to avoid generating images reminiscent of the renowned copyrighted photograph of the Afghan girl. Yet, as <cit.> note, merely banning the term “Afghan” does not prevent the model from recreating those images. In light of these issues, the development of techniques to detect and address such inadvertent memorizations has become crucial.



To address this, we first introduce a novel method for detecting memorized prompts. We've observed that for such prompts, the text condition consistently guides the generation towards the memorized solution, regardless of the initializations. This phenomenon suggests significant text guidance during the denoising process. As a result, our detection method prioritizes the magnitude of text-conditional predictions as its cardinal metric. Distinctively, memorized prompts tend to exhibit larger magnitudes than non-memorized ones, as showcased in <ref>. Unlike previous methods that query large training datasets with generated images <cit.> or assess density across multiple generations <cit.>, our strategy offers precise detection without adding any extra work to the existing generation framework and even without requiring multiple generations. In terms of efficacy, our method achieves an AUC of 0.960 and a TPR@1%FPR of 0.760 in under 2 seconds. In contrast, the baseline method <cit.> demands over 39 seconds, even though it registers an AUC of 0.934 and a TPR@1%FPR of 0.523. Such efficiency empowers model owners to halt generations early and initiate corrective measures promptly upon detecting a memorized prompt.

Building on our discoveries, we devise a strategy to highlight the influence of each token in driving memorization, aiming to pinpoint the specific trigger tokens responsible for it. Following the intuition that removing these trigger tokens should neutralize memorization, we anticipate a corresponding reduction in the magnitudes of text-conditional predictions. Thus, by evaluating the gradient change for every token when minimizing text-conditional prediction magnitudes, we discern the relative significance of every token to memorization. Armed with this tool, model owners can provide constructive feedback to users, guiding them to identify, modify, or omit these pivotal trigger tokens, effectively reducing the propensity for memorization. In contrast to earlier work like <cit.>, where trigger tokens were discerned manually from training data or by experimenting with various token combinations, our approach stands out for its automated nature and computational efficiency.

Lastly, we introduce mitigation strategies to address memorization concerns. Catering to both inference and training phases, we present model owners with a choice of two distinct tactics. For inference, we suggest using a perturbed prompt embedding, achieved by minimizing the magnitude of text-conditional predictions. During training, potential memorized image-text pairs can be screened out based on the magnitude of text-conditional predictions. Our straightforward approaches ensure a more consistent alignment between prompts and generations, and they effectively reduce the memorization effect when benchmarked against baseline mitigation strategies.



§ RELATED WORK



  
Membership Inference.
The membership inference attack <cit.> aims to determine if a particular data point was used in the training set of a model. Traditional studies on membership inference <cit.> have predominantly focused on classifiers. 
An attacker can utilize losses or confidence scores as a metric. This is because data points from the training set typically exhibit lower losses or higher confidence scores than the unseen data points during inference due to overfitting. In a parallel development, recent 
works <cit.> have extended membership inference to diffusion models. These methodologies involve introducing noise to a target image and subsequently verifying if the predicted noise aligns closely with the induced noise.



  
Training Data Extraction. 
<cit.> demonstrate that diffusion models memorize a subset of their training data, often producing the training image verbatim. Building on this fact, <cit.> introduce a black-box data extraction attack designed for diffusion models. This approach involves generating a multitude of images and subsequently applying a membership inference attack to assess generation density. Notably, they observe that memorized prompts tend to produce nearly identical images across different seeds, leading to high density. This strategy bears resemblance to the pipeline used by <cit.>, who successfully extract training data from large language models with over a billion parameters. Additionally, they discover that larger models are more susceptible to data extraction attacks compared to their smaller counterparts.



  
Diffusion Memorization Mitigation.
Recent research by <cit.> presents a method for training diffusion models using corrupted data. In their study, they demonstrate that their proposed training algorithm aids in preventing the model from overfitting to the training data. Their approach involves introducing additional corruption prior to the noising step and subsequently calculating the loss on the original input image. In a separate study, <cit.> delve into various mitigation strategies, with a focus on altering the text conditions. As a notable example, by inserting random tokens into the prompt or integrating random perturbations into the prompt embedding, they alleviate the memorization concern while preserving a high-quality generation output.



§ DETECT MEMORIZATION EFFICIENTLY



 §.§ Preliminary

We begin by defining the essential notation associated with diffusion models <cit.>. For a data point x_0 drawn from the real data distribution q(x_0), a forward diffusion process comprises a fixed Markov chain spanning T steps, where each step introduces a predetermined amount of Gaussian noise. Specifically:

    q(x_t | x_t-1) = 𝒩(x_t; √(1-β_t)x_t, β_t𝐈),  for  t ∈{1, ..., T},

where β_t ∈ (0, 1) is the scheduled variance at step t. The closed-form for this sampling is 

    x_t = √(α_t)x_0 + √(1 - α_t)ϵ,

where, α_t = ∏_i=1^t(1-β_t).

In the reverse diffusion process, a Gaussian vector x_T ∼𝒩(0,1) is denoised to map to an image x_0 ∈ q(x). At each denoising step, a trained noise-predictor ϵ_θ anticipates the noise ϵ_θ(x_t) that was added to x_0. Based on <ref>, the estimation of x_0 can be formulated as:

    x̂_0^t = x_t-√(1-α_t)ϵ_θ(x_t)/√(α_t).

Then, we can predict x_t-1 as:

    x_t-1 = √(α_t-1)x̂_0^t + √(1-α_t-1)ϵ_θ(x_t).




Text-conditional diffusion models, such as Stable Diffusion <cit.>, employ classifier-free diffusion guidance <cit.> to steer the sampling process. Given a text prompt p, its embedding e_p=f(p) is computed using a pre-trained CLIP text encoder f(·) <cit.>. In the reverse process, the conditional sampling adheres to <ref> and <ref>, but the predicted noise ϵ_θ(x_t) is changed to:

    ϵ_θ(x_t, e_∅) + s (ϵ_θ(x_t, e_p) - ϵ_θ(x_t, e_∅)_text-conditional noise prediction),

where, e_∅ represents the prompt embedding of an empty string, and s determines the guidance strength, controlling the alignment of the generation to the prompt. We refer to the term ϵ_θ(x_t, e_p) - ϵ_θ(x_t, e_∅) as the text-conditional noise prediction for future reference.



 §.§ Motivation

When provided with the same text prompt but different initializations, diffusion models can generate a diverse set of images. Conversely, when given different text prompts but the same initialization, the resulting images often display semantic similarities. These similarities include analogous layouts and color themes, as demonstrated in Appendix <ref>. Such a phenomenon might arise when the final generation remains closely tied to its initialization, and the textual guidance is not particularly dominant. This observation is consistent with findings from <cit.>, suggesting that one can trace the origin to the initial seed even without knowing the text condition.

Interestingly, when it comes to memorized prompts, the initialization appears to be irrelevant. The generated images consistently converge to a specific memorized visualization. This behavior implies that the model might be overfitting to both the prompt and a certain denoising trajectory, which leads to the memorized image. Consequently, the final image deviates substantially from its initial state.

These insights provide a foundation for a straightforward detection strategy: scrutinizing the magnitude of text-conditional noise predictions. A smaller magnitude signals that the final image is closely aligned with its initialization, hinting that it is likely not a memorized image. On the other hand, a larger magnitude could indicate potential memorization. The correlation between magnitude and memorization is depicted in <ref>.



 §.§ An Effective Detection Method


Following the intuition above, we introduce a straightforward yet effective detection method centered on the magnitude of text-conditional noise predictions. For a prompt embedding p and a sampling step of T, the detection metric is defined as

    d = 1/T∑_t=1^Tϵ_θ(x_t, e_p) - ϵ_θ(x_t, e_∅)_2.

Memorization is then identified if the detection metric falls beneath a tunable threshold γ.

In practice, we also find that even the detection metric of a single generation can provide a strong signal of memorization. Consequently, our method remains effective and reliable with the number of generations restricted to 1. In contrast, earlier studies, such as those examining generation density over a large number of generations <cit.>, require the simultaneous generation of multiple images, with some cases necessitating over a hundred generations. This might impose an extra computational burden on the service provider by generating more images than the user requested. Moreover, another method presented in <cit.> identifies memorized prompts by directly comparing the generated images with the original training data. Unlike this method, our approach allows a third party to use the detection method without needing access to the large training dataset, thereby protecting training data privacy.

Another distinct advantage of our approach is its adaptability in calculating the detection metric. Strong detection does not mandate collecting the metric from all sampling steps. Based on our empirical findings, even when the metric is collated solely from the first step, reliable detection remains attainable. This efficiency enables model owners to identify memorized prompts promptly. By stopping generation early, they can then opt for post-processing, like declining the generated output or reinitializing the generation with corrective strategies in place.








 §.§ Experiments





  
Experimental Setup.
To evaluate our detection method, we use 500 memorized prompts identified in <cit.> for Stable Diffusion v1 <cit.>, where the SSCD similarity score <cit.> between the memorized and the generated images exceeds 0.7. The memorized prompts gathered in <cit.> include three types of memorization: 1) matching verbatim: where the images generated from the memorized prompt are an exact pixel-by-pixel match with the original paired training image; 2) retrieval verbatim: the generated images perfectly align with some training images, albeit paired with different prompts; 3) template verbatim: generated images bear a partial resemblance to the training image, though variations in colors or styles might be observed.

Additionally, we use another 2,000 prompts, evenly distributed from sources LAION <cit.>, COCO <cit.>, Lexica.art <cit.>, and randomly generated strings. For this set of prompts, we assume they are not memorized by the model. All generations employ DDIM <cit.> with 50 inference steps.

In our comparison, we use the detection method from <cit.> as a baseline. This method determines memorization by analyzing generation density, computed using the pairwise ℓ_2 distance between non-overlapping tiles. While <cit.> utilizes the ℓ_2 distance in pixel space, we introduce an additional baseline that calculates the distance in the SSCD feature space <cit.>. This adjustment is inspired by <cit.>, who underscore the effectiveness of SSCD. As a deep learning-informed distance metric, SSCD offers enhanced resilience to particular augmentations, like color shift — a critical advantage when the training image is only partially memorized.

We use the area under the curve (AUC) of the receiver operating characteristic (ROC) curve and the True Positive Rate at the False Positive Rate of 1% (TPR@1%FPR) as metrics. Meanwhile, we report the running time in seconds with a batch size of 4 on a single NVIDIA RTX A6000.



  
Results.
In <ref>, we display a density plot comparing the detection metrics for the memorized prompts against the non-memorized ones, calculated over 50 steps with 4 generations per prompt. The distribution of memorized prompts is bimodal. This dichotomy stems from the fact that the template verbatim scenario often exhibits a slightly smaller metric than the matching verbatim scenario, given that the memorization occurs only partially.

In <ref>, we highlight the balance between the precision and efficiency of our proposed method. Our method is able to achieve very strong detection performance. When generating 32 images and using the metrics from the first 10 steps, our method is able to achieve an AUC of 0.999 and TPR@1%FPR of 0.988. Remarkably, even when operating with a single generation, our method can achieve TPR@1%FPR of 0.760 from the very first step within merely 0.2 seconds. This feature provides a significant advantage in terms of time and computational resource savings, allowing model operators the flexibility to terminate generation early if necessary. In contrast, the baseline methods show noticeably reduced detection capability. In particular, the baseline methods can only achieve relatively high detection accuracy when generating more than 16 images and relying on the image generation from the final step, where it requires at least 40 seconds. Yet, in real-world applications, service providers like Midjourney or DALL-E 2 <cit.> typically generate a mere 4 images concurrently for each prompt.

Interestingly, our method surpasses the baselines in speed even when using the metric with equivalent generations and steps. This superiority emerges since our method doesn't need to decode latent noise into image space and perform subsequent calculations.



§ MITIGATE MEMORIZATION





 §.§ A Straightforward Method to Detect Trigger Tokens

As observed by <cit.>, certain words or tokens in memorized prompts play a significant influence on the generation process. Even when only these specific “trigger tokens” are present in the prompt, the memorization effect remains evident. One potential approach to identify these trigger tokens involves probing with various n-gram combinations to discern which combinations induce memorization. However, this heuristic becomes notably inefficient, particularly when the prompt contains a vast number of tokens. Our earlier observations offer a more streamlined method for discerning the significance of each token in relation to memorization: by checking the magnitude of the change applied to each token while minimizing the magnitude of text-conditional noise prediction. A token undergoing substantial change suggests its crucial role in steering the prediction; conversely, a token with minimal change is less important.

Given a prompt embedding e of prompt p with N tokens, we form the objective of the minimization problem as:

    ℒ(x_t, e) = ϵ_θ(x_t, e) - ϵ_θ(x_t, e_∅)_2.

We then determine the significance score for each token at position i ∈ [0, N-1] as:

    SS_e^i = 1/T∑_t=1^T∇_e^iℒ(x_t, e)_2.


In <ref>, we display generations with top-2 significant tokens highlighted. The green arrow in the figure emphasizes that altering these significant tokens can substantially diminish the memorization effect. Some trigger tokens, including symbols or seemingly trivial words, are challenging to identify manually. Consequently, this insight offers model owners a practical tool: advising users to rephrase or exclude the trigger tokens before initiating another generation. In contrast, as indicated by the white arrow, alterations to the less significant tokens fail to effectively counter the memorization effect, even when extensive changes are made. Even by renaming the play or removing the artist's name, memorization remains evident.



 §.§ An Effective Inference-time Mitigation Method

A direct approach to mitigation without any supervision is to adjust the prompt embedding by minimizing <ref>. Optimizing over all time steps is computationally intensive. However, we observe that minimizing the loss at the initial time step indirectly results in smaller magnitudes in subsequent time steps, effectively mitigating memorization. Thus, a perturbed prompt embedding, e^*, is obtained at t=0 by minimizing <ref>. We also apply early stopping once the loss reaches a target value, l_target, to keep the embedding close to the original meaning.



 §.§ An Effective Training-time Mitigation Method

During the training of diffusion models, memorization often arises with duplicate data points due to overfitting <cit.>. Building on our earlier observation, if the model overfits or closely memorizes a data point, the magnitude of the text-conditional noise prediction might exceed typical values. Therefore, a straightforward mitigation method is to exclude the sample from the current mini-batch if this magnitude surpasses a predetermined threshold, τ, thereby not computing the loss on that sample. Given that the model has previously seen the sample during training, this exclusion is unlikely to significantly impact model performance.

However, this method introduces an additional computational cost during training. To compute the text-conditional noise prediction, an extra forward pass for ϵ_θ(x_t, e_∅) is needed. During typical training, only ϵ_θ(x_t, e_p) is computed. Empirically, this results in an approximately 10% increase in training time.





 §.§ Experiments



  
Experimental Setup. To evaluate the effectiveness of mitigation strategies, we adopt a setup similar to that in <cit.>. Specifically, we fine-tune the Stable Diffusion model using 200 LAION data points, each duplicated 200 times, to serve as memorized prompts. In addition, we introduce 120,000 distinct LAION data points to ensure that the model retains its capacity for generalization. 

For performance metrics, we compute the SSCD similarity score <cit.> to gauge the degree of memorization by comparing the generation to the original image. Additionally, the CLIP score <cit.> is used to quantify the alignment between the generation and its corresponding prompt. Our experiments encompass 5 distinct fine-tuned models, each embedded with different memorized prompts, and the results are averages over 5 runs with different random seeds.

In our evaluation of the proposed method, we test 5 distinct target losses l_target, ranging from 1 to 5, for inference-time mitigation. We use Adam optimizer <cit.> with a learning rate of 0.05 and at most 10 steps. Simultaneously, we investigate 5 different thresholds τ, spanning from 2 to 6, for training-time mitigation. For comparison, we use the most effective method from <cit.>, random token addition (RTA), as the baseline, which inserts 1, 2, 4, 6, or 8 random tokens to the prompt.





  
Results.
In <ref>, we present the results of our inference-time mitigation, while <ref> details the outcomes for training-time mitigation. Our proposed techniques successfully mitigate the memorization effect and, importantly, offer a more favorable CLIP score trade-off compared to RTA. Higher target losses l_target or thresholds τ tend to enhance the model's alignment with the prompt but can result in a less pronounced mitigation effect. In practice, model owners can select the optimal hyperparameter based on their desired balance between mitigation efficacy and generation alignment.

In <ref>, we display a selection of qualitative results, setting l_target=3 and τ=4 for our method, while adding 4 random tokens as a baseline strategy. Our proposed strategy is effective in mitigating the memorization effect while also ensuring that the generated content aligns closely with the prompt. 
In contrast, the baseline method often struggles to counteract memorization and occasionally produces images with undesirable additions.



§ LIMITATIONS AND FUTURE WORK

Our detection strategy employs a tunable threshold to detect memorized prompts. This requires model owners to first compute metrics over non-memorized prompts and then select an empirical threshold based on a predetermined false positive rate. However, the outcomes generated by this detection approach lack interpretability. In the future, the development of a method producing interpretable p-values could significantly assist model owners by providing a confidence score that quantifies the likelihood of memorization, thereby augmenting the transparency and trustworthiness of the detection process.

Our proposed mitigation strategies effectively tackle the memorization problem, albeit with minor computational overheads for model owners. The inference-time mitigation approach requires additional GPU memory due to the optimization process, and in practice, it takes at most 6 seconds with our setup. On the other hand, the training-time mitigation extends the training period by roughly 10%. For context, while naive fine-tuning in our tests takes approximately 9 hours, our mitigation strategy extends this to about 10 hours. However, we argue that these added costs are reasonable given the significance of protecting training data privacy and intellectual property. Furthermore, when combined with our detection method, model owners need only deploy the inference-time mitigation when a memorized prompt is detected, thereby minimizing its use.



§ CONCLUSION

In this paper, we introduced a new approach to detect memorization in diffusion models by leveraging the magnitude of text-conditional noise predictions. Remarkably, our approach attains high precision even when using a limited number of generations per prompt and a limited number of sampling steps. Furthermore, we provide an explanatory tool to indicate the significance score of individual tokens in relation to memorization. To conclude, our paper presents both inference-time and training-time mitigation strategies. These not only effectively address the memorization concern but also maintain the superior generative performance of the model.



§ REPRODUCIBILITY STATEMENT

We have detailed all essential hyperparameters used in our experiment setup in the main body. Our experiments were conducted using widely available computing resources and open-source software. All referenced models and datasets in this paper are publicly accessible. Furthermore, we have included the code necessary to reproduce our results in the supplementary material. 



§ DISCLAIMER

This research required the use of some images from the LAION 5B dataset for analysis purposes. After our use and deletion of such images, it was discovered that LAION 5B contained exploitative images of children. While we have no reason to believe that such harmful images were present in our subset, we acknowledge child safety is a serious issue and believe it is important to be transparent about the dataset's use and its associated ethical considerations.


iclr2024_conference



§ APPENDIX










 Paper Title: Enhanced Self-Checkout System for Retail Based on Improved YOLOv10.












































Enhanced Self-Checkout System for Retail Based on Improved YOLOv10
    
  
       Lianghao Tan                            Shubing Liu                                 
        
       Arizona State University                University of North Carolina at Chapel Hill 
 
       

       Jing Gao                                Xiaoyi Liu                                  

       University of Minnesota                 Arizona State University                    

       
 
       Linyue Chu                              Huangqi Jiang                               

       University of California, Irvine        Case Western Reserve University
  

    August 5, 2024
=======================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================





§ ABSTRACT 

With the rapid advancement of deep learning technologies, computer vision has shown immense potential in retail automation. This paper presents a novel self-checkout system for retail based on an improved YOLOv10 network, aimed at enhancing checkout efficiency and reducing labor costs. We propose targeted optimizations to the YOLOv10 model, by incorporating the detection head structure from YOLOv8, which significantly improves product recognition accuracy. Additionally, we develop a post-processing algorithm tailored for self-checkout scenarios, to further enhance the application of system. Experimental results demonstrate that our system outperforms existing methods in both product recognition accuracy and checkout speed. This research not only provides a new technical solution for retail automation but  offers valuable insights into optimizing deep learning models for real-world applications. 





§ INTRODUCTION





 §.§ Research Background

Driven by the digital revolution, traditional retail stores face numerous challenges and opportunities for transformation. As consumer demands diversify and shopping behaviors rapidly evolve, the inefficiency of traditional checkout processes in retail stores has become apparent, often resulting in poor customer experiences. Especially during peak times, customers may endure long waits to pay, negatively impacting their shopping experience and limiting the store's capacity and potential for sales growth.

With the rapid advancement of technology, the wave of Artificial Intelligence (AI) is profoundly transforming a wide range of industries, including retail<cit.>. Both 2-D and 3-D vision models are widely used in different scenarios<cit.> and real-time tracking detection can be used on both manned and unmanned objects<cit.>. In the realm of AI, Machine Learning has become a powerful tool for tackling complex tasks and enhancing efficiency. Deep Learning and Computer Vision, two significant subsets of Machine Learning, have gained attention in recent years for their precise handling and analysis of visual data even in difficult and remote scenarios<cit.>. Convolutional Neural Networks (CNNs), a landmark architecture in Deep Learning, have demonstrated excellent performance in computer vision tasks and are extensively used for image classification, object detection, and segmentation. In the retail sector, CNNs can significantly enhance product recognition accuracy and inventory management, and elevate customer experience and market analysis to new heights.

Before the advent of Deep Learning, the field of Object Detection primarily relied on traditional manual feature engineering techniques while deep leaning advanced the object detection and brought the benefit to retails and consumers <cit.>. Solutions from this era typically followed a two-stage process: feature extraction and classification. Besides the advancement of algorithms for exploring data samples <cit.>, techniques from image processing and computer vision were used to manually construct and extract prominent features from images, such as Haar feature sets and Scale-Invariant Feature Transform (SIFT)<cit.>. These designed features were then fed into classifiers to identify objects within images. However, the effectiveness of this approach heavily depended on the quality of the feature engineering and often fell short in complex and variable image backgrounds and scenes. With the rise of Deep Learning, especially the widespread adoption of CNNs, a significant leap occurred in object detection. CNNs, with their robust automatic feature-learning capabilities, eliminated the need for cumbersome manual feature design<cit.>. They could directly learn hierarchical and rich feature representations from raw image data, greatly enhancing the accuracy and efficiency of object detection. This phase saw the emergence of several iconic object detection algorithms such as Faster R-CNN, You Only Look Once(YOLO), and Single Shot MultiBox Detector(SSD), each marked by their unique design philosophies and exceptional performance, becoming hallmarks of the Deep Learning era in object detection.




 §.§ Current Study


Ensuring the accuracy of both the products being sold and the total price is crucial during the goods selling process. Traditional transaction processes often rely on manual operations, which are associated with subjectivity, human error, and high labor costs. Although the development of automated checkout machines that scan product barcodes has improved efficiency, these systems still require customers to scan items individually and remain prone to errors. To further reduce labor costs and enhance product identification accuracy and efficiency, object detection technology offers an effective solution. By improving automated checkout machines through the integration of object detection technology and the creation of a conducive identification environment, these new systems can recognize and count multiple items simultaneously<cit.>. With additional post-processing, these machines can then provide an accurate total price.






The YOLO series has advanced rapidly in recent years, achieving an effective balance between computational cost and detection performance. Extensive research has explored various aspects of YOLO's architectural design, optimization objectives, data augmentation strategies, etc. On May 23, 2024, a new iteration in the series, YOLOv10, was introduced. In our study, we focused on targeted optimizations of YOLOv10 on a reset retail product dataset (RPC). The structure of this study is as follows: First, we utilized YOLOv8 and YOLOv10 to create the MidState-YOLO network. We then modified the traditional C2f model using Dual Convolutional kernels (DualConv) to develop the C2f_Dual module, which replaced  the C2f module before the Spatial Pyramid Pooling layer (SPPF). Additionally, we integrated the Efficient Multi-Scale Attention module to enhance the network's ability to accurately locate and recognize retail products. The final developed model is the MidState-YOLO-ED network.





To validate the effectiveness of the improvement strategies, we conducted the comprehensive experimental verifications and confirmed the effectiveness of the improved algorithms by exploring the impact of different improvement strategies on the model's recognition performance. The experimental results demonstrated that the proposed improvement strategies could significantly enhance the model’s recognition accuracy while maintaining the parameters and computational load within a manageable range.



§ RELATED WORK




 §.§ Traditional Object Detection Algorithms


In the field of computer vision, object detection remains a core technology that continuously attracts the attention of researchers. Traditional deep learning methods for object detection are primarily based on CNNs. Among them, Faster R-CNN, YOLO, and SSD  stand as three landmark achievements in the domain, each advancing the technology in its unique way.

Faster R-CNN represents a significant improvement within the R-CNN series. It not only retains the advantages of the series in object detection but significantly enhances accuracy and speed through a series of innovations<cit.>. By introducing the Region Proposal Network (RPN), it automatically extracts potential target areas, effectively reducing the redundant computations brought by traditional sliding window methods. The RPN network efficiently utilizes small batch samples for training, combining with an anchor box and true bounding box matching strategy, to rapidly generate candidate regions, completely eliminating the time-consuming and complex selective search process. Furthermore, Faster R-CNN implements the idea of feature sharing, allowing the convolutional layers to be shared for both feature extraction and region proposal tasks, which could significantly improve  the efficiency and accuracy on detection

YOLO is a regression-based object detection method that simplifies the object detection process. Through a single forward network pass, it directly predicts the locations and categories of objects in the image, treating detection as a regression problem. By gridding the image and predicting the confidence levels of objects within each grid, YOLO achieves rapid detection.

SSD merges the efficiency of Faster R-CNN with the simplicity of YOLO, aimed at achieving efficient and accurate object detection. It adopts a single-shot multibox detection mechanism that directly outputs the location and class information of objects in one forward pass, effectively eliminating the computational redundancies brought by sliding windows and significantly increasing detection speed. Another highlight of SSD is its multi-scale feature fusion strategy, which flexibly handles targets of various sizes, enhancing the robustness and accuracy of detection. In its implementation, SSD pre-defines multiple sets of candidate boxes of different scales and aspect ratios, carefully adjusted during the training phase to cover potential target areas in the image. By extracting features through a shared CNN and then detecting objects in parallel across multiple feature maps, SSD fully utilizes different levels of feature information. This design not only speeds up detection but also enhances the model's sensitivity to various scale targets. Furthermore, SSD employs the concept of default boxes, aiming at further optimizing the detection of potential target areas which are not explicitly covered by the candidate boxes. By training the default boxes and their corresponding classifiers, SSD could  covers the targets in the image more comprehensively, further improving the thoroughness and accuracy of detection.



 §.§ Development of the YOLO Series


Since Joseph Redmon and his collaborators first introduced the YOLO series in 2016, it has made significant strides in real-time object detection by transforming the task into a regression problem. With the rapid development of computer vision technology, the YOLO series has continuously evolved, introducing new technologies and methods to address many issues found in the original version and gradually improving detection performance.

YOLOv1 was groundbreaking in converting the object detection task into a single regression problem. It divided the input image into multiple grids and predicted bounding boxes and class information within each grid, achieving fast and accurate object detection. However, YOLOv1 had limitations in detecting small and overlapping objects. YOLOv2 made several improvements to address these shortcomings. It introduced the anchor boxes mechanism to enhance the accuracy of bounding box predictions and adopted a new backbone network, Darknet-53, to improve feature extraction capabilities. Additionally, it incorporated multi-scale training strategies to enhance the model's generalization ability. These improvements boosted both the detection speed and accuracy of YOLOv2.

YOLOv3 further expanded the capabilities of the YOLO series, introducing multi-scale feature fusion strategies and employing a Feature Pyramid Network (FPN) design. This enhancement enabled YOLOv3 to better capture contextual information within images, thereby improving its ability to detect objects of various sizes.

YOLOv4 and YOLOv5, building on the strengths of their predecessors, underwent further optimizations and improvements in model architecture, feature extraction, and loss functions, significantly enhancing the accuracy and speed of object detection<cit.>.

YOLOv8, developed by Ultralytics after YOLOv5, represents the next generation in the YOLO series, built on the success of previous models<cit.>. It features a more efficient backbone network and detection head design, incorporating new convolutional blocks and activation functions to improve feature extraction and bounding box prediction accuracy. Additionally, YOLOv8 offers more convenient installation and operational methods, making it easier for users to deploy and utilize on various hardware platforms.

As the latest member of the YOLO series, YOLOv10 has made groundbreaking advancements in end-to-end and real-time aspects. It introduces a training strategy without NMS, utilizing a dual-label assignment mechanism to eliminate the inference delays traditionally caused by NMS.



 §.§ YOLOv8 and YOLOv10 Network Structures and Their Differences


The network architecture of YOLOv8 includes three main components: Backbone, Neck, and Head. YOLOv8 employs an enhanced CSPDarknet as its backbone network, which is optimized based on the CSP design to better balance computational complexity and feature expressiveness. Specifically, CSPDarknet splits the feature map at each stage into two parts: one part undergoes deep processing through a Dense Block, and the other part is directly concatenated with the output of the Dense Block. This design not only reduces computational costs but also enhances gradient flow and feature reuse, benefiting the overall performance of the model. YOLOv8 also introduces the advanced activation function SiLU (Swish), replacing the traditional Leaky ReLU. SiLU maintains non-linear characteristics while facilitating better gradient flow, aiding the model in converging faster and achieving higher accuracy during training.

The Neck of the network utilizes a Path Aggregation Network (PANet), which builds upon the Feature Pyramid Network (FPN) by adding an extra bottom-up path to further enhance the flow and fusion of information between feature layers. PANet's design includes a bottom-up feature extraction path, a top-down semantic feature propagation path, and an additional bottom-up path for feature enhancement. At each level, features from different paths are fused either by element-wise addition or concatenation, thus improving the network's capability to detect targets of varying scales.

Unlike traditional YOLO models, YOLOv8 adopts an Anchor-Free point detection Head, which simplifies the model architecture and reduces computational overhead. The Anchor-Free method directly predicts the center points of objects and their attributes (such as size, orientation, etc.), avoiding the limitations and complexity of predefined anchor boxes. This change not only speeds up inference but also enhances the model's ability to detect small and densely packed objects, particularly suited for edge computing scenarios.

YOLOv8 also incorporates the C2f (Cross-stage Partial Convolution with two types of fusion) building block, which, by enhancing feature extraction and fusion capabilities, further improves the model's detection accuracy<cit.>. The C2f building block combines the advantages of the CSP design and innovates in feature fusion, allowing the model to better capture fine-grained details and complex patterns.

The network structure of YOLOv10 is broadly similar to that of YOLOv8, with enhancements in the backbone network, Neck structure, Head design, and an innovative Non-Maximum Suppression (NMS)-free training strategy. YOLOv10, during training, incorporates a dual-label assignment strategy, combining one-to-many and one-to-one allocations. One-to-many allocation improves recall, while one-to-one ensures precision. This combination maintains consistency between training and inference, eliminating the need for NMS during inference. Removing the NMS step significantly reduces inference time and simplifies the inference process, which is crucial for real-time applications and edge devices.

YOLOv10 optimizes various components of the model architecture to minimize computational overhead while enhancing performance. For efficiency, YOLOv10 employs depthwise separable convolutions to reduce the computational cost of the classification head and proposes spatial-channel decoupled downsampling for more efficient downsampling. YOLOv10 also uses large kernel depthwise convolutions in the compact inverted blocks (CIB) at deeper stages to expand the receptive field and enhance feature extraction capabilities, integrating a Partial Self-Attention (PSA) module to improve global representation learning with minimal overhead, thereby enhancing detection accuracy. These improvements make YOLOv10 superior in terms of real-time performance and accuracy.



 §.§ Loss Functions


In YOLOv8, the decoupled head is used to separate the classification and detection heads. The classification branch employs the Binary Cross-Entropy (BCE) loss, which is formulated as follows:


    Loss_BCE = -w[y_nloglog x_n + (1-y_n)loglog(1-x_n)]
                                                                                

                         
where “w” represents the weight; “y” represents the label, and “x” represents the predicted value generated by the model<cit.>.
For the regression branch, it combines the use of Distributional Focal Loss (DFL) and CIoU loss. The DFL function is designed to emphasize the expansion of probability values around the object “y”. Its equation is as follows:


    Loss_DF=-[(y_n+1-y) log log y_i+1-y/y_(i+1)-y_i
    
        + (y-y_n ) log log y-y_i/y_i+1-y_i]
  

CIoU loss introduces an influence factor to the DIoU loss by considering the aspect ratio between the predicted bounding box and the actual bounding box. The corresponding equation is as follows:

    Loss_CIoU=1-IoU+d^2/c^2 + v^2/(1-IoU)+v
  

where IoU measures the overlap between the predicted bounding box and the true bounding box; “d” is the Euclidean distance between the centers of the predicted and true bounding boxes, and “c” is the diagonal length of the smallest enclosing box that contains both the predicted and true bounding boxes. Additionally, “v” represents the parameter quantifying the consistency of the aspect ratio, defined by the following equation:

    v=4/π^2 (arctan w_gt/h_gt -arctan w_p/h_p )^2
  

where “w” represents the weight of the bounding box; “h” represents the height of the bounding box; “gt” indicates the ground truth, and “p” represents the prediction<cit.>.



§ THE IMPROVED MIDSTATE-YOLO-ED NETWORK



 §.§ Integration of YOLOv8 and YOLOv10

One of the most distinct features of YOLOv10 compared to its predecessors is the elimination of Non-Maximum Suppression (NMS), achieved by introducing a consistent dual-assignment strategy. This strategy involves calling the loss function calculation method of YOLOv8 twice, summing the results, and returning them. This approach aims to address the issue of redundant predictions in post-processing, aligning closer to the end-to-end direction of the RT-DETR model. Unfortunately, this modification led to a decrease in accuracy for many datasets in practical applications. To avoid loss of precision, the prediction head of YOLOv10 was reverted to that of YOLOv8<cit.>.

To significantly reduce computational redundancy and achieve a more efficient architecture, YOLOv10 employs a lightweight classification head, Spatial-Channel Decoupled Downsampling (SCDown), and a rank-based block design (C2fUIB). SCDown first adjusts the channel dimensions using point-wise convolution, followed by spatial downsampling using depthwise convolution to reduce the number of parameters. However, some information loss occurs during the downsampling process in experiments, which, although reducing latency, does not ensure performance gains.

Due to the modifications in this study reverting two core components of YOLOv10 back to YOLOv8 status, and integrating both versions, the current model has been named MidState-YOLO. Additionally, the model cleverly integrates EMA attention and the C2f-Dual convolution design, ultimately naming the final model as the MidState-YOLO-ED network.



 §.§ Integration of EMA Attention

To further enhance the expressive capability of the MidState-YOLO network and establish long and short dependency relationships, Efficient Multi-Scale Attention (EMA) has been integrated into the Neck network. EMA is a parallel attention mechanism used in computer vision tasks to improve model performance and processing speed. Unlike traditional Convolutional Neural Networks (CNNs), EMA adopts a parallel structure to handle input data. This parallel convolution allows faster training of models when dealing with large-scale data and enhances accuracy by enabling parallel processing of features at different scales. In figure x, the divided groups are represented by "g," while "X Avg Pool" denotes one-dimensional horizontal global pooling, and "Y Avg Pool" represents one-dimensional vertical global pooling respectively.
The formula for the average pooling operation is as follows, where Xc(i,j) represents the element at position (i,j):

    Z_c = 1/H × W∑_j^H∑_i^WX_c ( i, j  )
   

The input to EMA is first grouped and reshaped, redistributing part of the channel dimensions to the batch dimension. This is followed by further subdivision of the channel dimension into multiple sub-features to preserve key information in each channel and optimize the distribution of spatial semantic features<cit.>.



This structure contains two main parallel branches: one branch performs one-dimensional global pooling operations to encode global information, while the other branch performs feature extraction through a 3x3 convolution. The output features from these two branches are modulated through a sigmoid function and normalization processes, and then merged through a cross-dimensional interaction module to capture pixel-level pairwise relationships. Finally, the sigmoid-modulated output feature maps are used to enhance or weaken the original input features, thus achieving a more refined and effective feature representation. Therefore, EMA not only encodes inter-channel information to adjust the importance of various channels but also preserves precise spatial structural details within these channels.



 §.§ Lightweight Dual Convolution C2f-Dual Design

In YOLOv8 and YOLOv10, the C2f module integrates both low-level and high-level feature maps, facilitating the capture of gradient information flows. However, with the increasing number of layers in convolutional neural networks, semantic information in feature maps tends to be progressively extracted and aggregated, leading to redundancy in deep feature maps. Additionally, due to the weight-sharing mechanism of convolutional layers, convolutional kernel parameters are shared at different positions in the deep feature maps, further exacerbating redundancy. Bottleneck modules, composed of many complex convolutions, significantly increase parameter size and computational complexity.

To address this issue, the C2f-Dual convolution design, improved using DualConv, significantly reduces computational costs and the number of parameters while also enhancing precision. This improvement involves replacing the C2f modules before the Spatial Pyramid Pooling (SPPF) with C2f-Dual modules, as shown in Figure 1. This adaptation not only streamlines the network but also optimizes performance by ensuring that critical spatial and semantic features are efficiently processed and integrated.

DualConv is designed to build lightweight deep neural networks by combining 3×3 and 1×1 convolution kernels to process the same input feature map channels, optimizing information processing and feature extraction. In DualConv, the 3×3 convolution kernels are used to extract spatial features from the feature maps, capturing more spatial information, while the 1×1 convolution kernels integrate these features and reduce the model's parameters. Each group of convolution kernels processes a portion of the input channels independently before the outputs are merged, facilitating efficient flow and integration of information across different feature map channels.

Additionally, DualConv employs group convolution technology to efficiently arrange convolution filters. In group convolution, both input and output feature maps are divided into multiple groups, with each group’s convolution filters processing only a part of the corresponding input feature map. This arrangement allows different kernels within a group to process the same set of input channels in parallel, optimizing information flow and feature extraction efficiency while maintaining the network's representational capabilities.

Thus, replacing the bottleneck structures in C2f with DualBottleneck enriches gradient flow representation, enhances feature extraction capabilities, and reduces the diversity of false positives and false negatives in network learning. This makes it more suitable for retail commodity object detection scenarios.



§ EXPERIMENTAL RESULTS AND ANALYSIS



 §.§ Experimental Setup and Parameters

The hardware environment and software configurations used for the experiments are detailed in Table 1. During the model training process, the learning rate was set to 0.01, with optimization carried out using Stochastic Gradient Descent (SGD). The momentum parameter was set at 0.937, and the weight decay factor was 0.0005. The batch size used was 32, and the image size was 640x640 pixels. The comparative experiments were conducted over 30 epochs, while the ablation studies were carried out over 25 epochs.



 §.§ Dataset

This study employs a portion of the RPC dataset for training and validation. The RPC dataset, developed by Megvii Technology's Nanjing Research Institute, is currently the largest product recognition dataset available. It includes up to 200 different product categories and totals 83,000 images, realistically simulating retail environments and surpassing existing datasets in fidelity. Moreover, it effectively captures the fine-grained characteristics inherent in the Automatic Check-Out (ACO) problem.

The conceptual approach of this study may differ from that of the researchers who collected the RPC dataset<cit.>. When customers enter a store and place the items they wish to purchase on the checkout counter, an ideal ACO system would automatically identify each product and accurately generate a shopping list in one go, as shown in Figure 2. Thus, ACO is fundamentally a system designed to identify and count the occurrence of each item in any combination of products. Due to the continuous updating of a vast number of product categories, to avoid exhausting all possible combinations, a feasible solution adopted by the RPC dataset is to collect images of a single product type in a specific environment and reuse them in actual settlements. 

This research posits that there are multiple important metrics for assessing performance on Automatic Check-Out (ACO) tasks. To ensure accuracy and performance, the images used to train the ACO recognition system should mirror the actual retail checkout environments, which can indeed be simplified and stabilized. Additionally, initial models do not need to exhaust all product combinations to perform ACO tasks; instead, creating random groups of product combinations suffices.

Therefore, for this study, only the checkout configurations from the RPC dataset images are used. We randomly divided 30,000 images of checkout configurations into training, validation, and test sets in an 8:1:1 ratio. This approach aims to provide a realistic yet controlled set of data that reflects real-world ACO system operations while maintaining manageable complexity and variety in training scenarios.







 §.§ Evaluation Metrics

This paper employs Precision (P), Recall (R), mean Average Precision (mAP), number of parameters (Params), and Floating Point Operations (FLOPs) as evaluation metrics, with a set IoU threshold of 0.5. mAP@0.5 denotes the mean Average Precision when the IoU is set at 0.5, and mAP@0.5:0.95 indicates the mean Average Precision when the IoU ranges from 0.5 to 0.95, with a step size of 0.05. The floating point operations indicate the complexity of the algorithm. The specific meanings of other performance metrics are as follows:


    P = (TP/TP + FP) × 100
   


    R = (TP/TP + FN) × 100
   

Precision is the probability that a positive sample predicted by the model is indeed a positive sample, and recall is the probability that a positive sample in reality is predicted as positive by the model. These are two important metrics used to evaluate model performance. The expressions are as follows: 


    AP = ∫_0^1 p(r)dr
   


    mAP = 1/n_j ∑_j=1^n_jAP_j
   

In the formulas, TP (True Positives) refers to positive examples correctly identified as positive by the model; FP (False Positives) refers to negative examples incorrectly identified as positive; FN (False Negatives) refers to positive examples incorrectly identified as negative.

mAP represents the mean of the average precision across all object detection categories. AP is the average of precision values at different recall levels. The curve plotted with Precision (P) as the y-axis and Recall (R) as the x-axis is known as the PR curve. mAP is then calculated as the average area under the PR curves for all categories. n represents the number of instances in a given category; APj represents the detection precision for category j.



 §.§ Comparative Experiments and Results



The experimental results primarily organize the performance parameters of the trained algorithms, explaining the strengths and weaknesses of each algorithm based on these results, and analyzing the experimental data and actual detection effects. All experiments were conducted under the same configuration settings, and after training, the weight files generated by each algorithm were tested. The algorithms used in the comparative experiments include SSD (Single Shot MultiBox Detector), Faster-RCNN (Faster Regions with CNN features), YOLOv8-n, and YOLOv10-n. The experimental results are shown in Table 2.

The experimental results indicate that, compared to the SSD and Faster-RCNN algorithms, the YOLO series algorithms and the improvements introduced in this study exhibit superior detection performance. Additionally, since Faster-RCNN is a two-stage algorithm, its applicability is limited by its complexity and the extended duration required for detection, which makes its overall performance inferior to that of the YOLO series.

The comparative results of different algorithms demonstrate that the MidState-YOLO-ED algorithm excels in terms of the number of parameters and floating point operations, with only 3,288,096 parameters and 9.6 GFLOPs. This fully proves the excellent performance of the improved algorithm in terms of lightweight design. The algorithm can process image data quickly and accurately, making it suitable for scenarios requiring fast response. It is also more appropriate for operation in resource-constrained environments such as mobile devices and embedded systems, and it meets real-time requirements<cit.>.

Furthermore, key indicators such as recall and mAP for the MidState-YOLO-ED algorithm are higher than those of baseline algorithms, offering a new option for efficient and rapid object detection.




 §.§ Ablation Study

To investigate the extent of improvements from three modification schemes, ablation studies were conducted using YOLOv8-n and YOLOv10-n as baseline networks. These experiments were carried out on the experimental dataset without changing the software and hardware environment, with the only parameter change being the reduction of epochs to 25. As indicated in Table 3, the MidState-YOLO network, which integrates modules from YOLOv8 and YOLOv10, achieved a 23.2 percentage point increase in mAP compared to the original baseline network of YOLOv10-n. This suggests that combining two models can harness the advantages of both while avoiding some of their respective shortcomings, thereby enhancing model performance.



Using the lightweight dual convolution C2f_Dual module in the MidState-YOLO network significantly reduced the number of parameters, and all performance metrics showed improvements. The reason for this is the reduction in redundant information in the deep feature maps and a decrease in false positives and false negatives during network learning. The inclusion of the EMA module increased the mAP@0.5 to 84.4%, resulting in a 4.2% gain. This demonstrates that the EMA module effectively captures global information to learn richer semantic features, focusing the network model more on the overall context of retail product targets and enhancing model performance.

The final improved model, MidState-YOLO-ED, showed improvements in all evaluation metrics relative to YOLOv8-n, with precision and recall increasing by 2.3 and 1.6 percentage points, respectively, and mAP reaching 89%. Additionally, the number of parameters and floating point operations were significantly lower.



 §.§ Visualization of Results

After training the final improved model, MidState-YOLO-ED, for 30 epochs, the prediction results are displayed in Figure 3. The results demonstrate that the algorithm proposed in this paper achieves excellent detection performance while maintaining a lightweight framework.




§ CONCLUSION

In this paper, we presented an enhanced self-checkout system using an improved YOLOv10 network. The system significantly advances retail automation by optimizing checkout efficiency and minimizing labor costs. Our adaptations to the YOLOv10 model, integrating features from YOLOv8 and new post-processing algorithms, markedly improves product recognition accuracy, with our experiments demonstrating superior performance over existing systems. Broader applications in inventory control and customer service will be benefited by this study. Our study shows that AI-driven technologies will play a pivotal role in enhancing consumer experiences and operational efficiency.





IEEEtran



























 Paper Title: Contrastive Factor Analysis.




Ge-based Clinopyroxene series: first principles and  experimental local probe study
    João P. Araújo
    August 5, 2024
===================================================================================




§ ABSTRACT 
Factor analysis, often regarded as a Bayesian variant of matrix factorization, offers superior capabilities in capturing uncertainty, modeling complex dependencies, and ensuring robustness.
As the deep learning era arrives, factor analysis is receiving less and less attention due to their limited expressive ability. 
On the contrary, contrastive learning has emerged as a potent technique with demonstrated efficacy in unsupervised representational learning.
While the two methods are different paradigms, recent theoretical analysis has revealed the mathematical equivalence between contrastive learning and matrix factorization, providing a potential possibility for factor analysis combined with contrastive learning.
Motivated by the interconnectedness of contrastive learning, matrix factorization, and factor analysis, this paper introduces a novel Contrastive Factor Analysis framework, aiming to leverage factor analysis's advantageous properties within the realm of contrastive learning. 
To further leverage the interpretability properties of non-negative factor analysis, which can learn  disentangled representations, contrastive factor analysis is extended to a non-negative version.
Finally, extensive experimental validation showcases the efficacy of the proposed contrastive (non-negative)  factor analysis methodology across multiple key properties, including expressiveness, robustness, interpretability, and accurate uncertainty estimation.





§ INTRODUCTION

Before the era of deep learning, factor analysis (FA) <cit.> had received widespread attention and achieved impressive results.
Generally, FA is a statistical method used to identify underlying factors or latent variables that explain
patterns of correlations among observed variables. 
With its stochastic latent variables, FA often enjoys good properties for modeling complex dependence and enhancing robustness <cit.>. 
Meanwhile, its non-negative extensions, such as Poisson factor analysis <cit.>, can not only learn meaningful latent representation from high-dimension observation unsupervised but also can discover interpretable semantic concepts 
 <cit.>. 
With these appealing properties, FA and non-negative FA have been widely used in many applications, including text analysis <cit.> image representation learning <cit.>. 

With the deep learning era arriving <cit.>, FA is receiving less and less attention due to its limited expressiveness. 
Correspondingly, recent years have witnessed a surge of interest in self-supervised learning, such as contrastive learning (CL) <cit.>, owing to its remarkable capacity to learn high-quality representations unsupervised, leading to notable advancements in various fields <cit.>.
Generally, CL methodologies aim to extract discriminative features by training models to distinguish positive and negative samples. 
The distinction between FA and CL lies primarily in the latter's ability to harness effective training objectives <cit.> and expressive deep neural networks <cit.>. 


Given that FA and CL are regarded as two distinct research fields, integrating CL to enhance FA or vice versa might appear challenging. Fortunately, recent advancements in the theoretical underpinnings have provided insights into this integration.  Specifically, <cit.> leverages spectral contrastive loss to derive the mathematical equivalent of CL and matrix factorization (MF) <cit.>.

Considering the relationship between MF and FA <cit.>, the integration of FA with CL becomes feasible. 
Beyond feasibility, merging FA and CL offers numerous attractive potential benefits. 
In general, from the perspective of FA, enhancing FA with CL can alleviate its limitation in expressiveness, leading to richer and more informative representations <cit.>.
From the perspective of CL, considering FA's capacity to capture complex dependencies and its stochastic latent variables, incorporating CL with FA not only enhances performance but also facilitates improved uncertainty estimation and interpretability <cit.>.










 
Drawing inspiration from the mathematical equivalence established between MF, CL, 
and FA, as illustrated in Figure. <ref>, in this paper, we try to develop a Contrastive Factor Analysis (CFA) framework. 
Unlike FA, which directly models observation data, CFA follows CL methods to model the relation matrix, which represents the relationship between positive and negative samples.
More specifically, in CFA, the relation matrix is factorized into Gaussian latent variables. 
Considering the target matrix is non-negative, the CL can also be modified as a contrastive non-negative FA, which factorizes the relation matrix into gamma latent variables.
Compared with CFA, its non-negative extension can enjoy good properties for learning disentangled representation <cit.>. 

To approximate the posterior distribution of the latent variables, we further develop a variational inference network.
While it's straightforward to employ the Gaussian distribution to appropriate the posterior of Gaussian latent variables,  considering the non-reparameterizability of the gamma distribution, we instead utilize the Weibull distribution to approximate the posterior of gamma latent variables <cit.>. 
Finally, both models' parameters can be effectively trained within a variational inference framework <cit.> with reparameterize technique. 


To assess the properties of contrastive factor analysis, we conducted a series of experiments targeting various aspects. 
These properties encompass the attainment of more expressive representations, robustness against out-of-distribution data, interpretability, and uncertainty evaluation. 




The main contributions of this paper can be summarized as follows: 



  * 
Drawing inspiration from the interplay between contrastive learning, matrix factorization, and factor analysis, we introduce a novel contrastive (non-negative) factor analysis framework. 


  * To approximate the posterior distribution of latent variables, we propose two variational inference networks: Gaussian and Weibull variational inference network.


  *  Extensive experimental validation showcases the efficacy of the proposed contrastive factor analysis methodology across multiple key properties, including expressiveness, robustness, interpretability, and accurate uncertainty estimation.





§ PRELIMINARY ON CONTRASTIVE LEARNING



In this section, we briefly review the standard contrastive learning modules and the relationship between contrastive learning and matrix factorization.


  
Contrastive learning. 
In representation learning, the aim is to learn an encoder function  f: ℝ^D→ℝ^d that maps high-dimensional input data  x ∈ℝ^D to low-dimensional representations z ∈ℝ^d. 
To perform contrastive learning, we start by building pairs of positive samples ( x,x_ + ) by augmenting the same natural sample x̅∼𝒫( x̅),

where 𝒜( ·|x̅) represents the augmentation distribution. Additionally, negative samples x̅ are independently augmented samples, following the marginal distribution 𝒫( x ) = 𝔼_x̅𝒜( x |x̅).

The objective of contrastive learning is to encourage the model to map positive samples closer together in the feature space while pushing negative samples further apart.
A well-known CL objective is the InfoNCE loss <cit.>. 









    0.95ℒ_(f ) = -𝔼_x, x_+, {x_i^ - }_i = 1^Mlogexp( f( x )^Tf( x_ + ))/exp( f( x )^Tf( x_ + )) + ∑_i = 1^M exp( f( x )^Tf( x_i^ - ))


where {x_i^ - }_i = 1^M are M negative samples independently drawn from 𝒫(x). <cit.> proposed the spectral contrastive loss that is more amenable for the theoretical analysis: 

    0.95ℒ_(f) =  - 2𝔼_x,x + f( x )^Tf( x_ + ) + 𝔼_x,x + ( f( x )^Tf( x_i^ - ))^2





  
Contrastive learning as matrix factorization. 
Notably, <cit.>
 show that the spectral contrastive loss can be equivalently written as the following matrix factorization objective: 

    ℒ_ = A̅ - FF^T^2,   F_x,: = √(P( x )) f( x )



Here, A̅ = D^-1/2 A D^-1/2 denotes the normalized version of the co-occurrence matrix A ∈ℝ_+^N × N of all augmented samples x ∈𝒳 (assuming |𝒳| = N for ease of exposition): ∀ x, x' ∈𝒳, A_x, x' := 𝒫(x, x') = 𝔼_x̅[ 𝒜(x|x̅) 𝒜(x'|x̅) ].


 



§ CONTRASTIVE (NON-NEGATIVE) FACTOR ANALYSIS

This section furnishes a comprehensive overview of the proposed contrastive (non-negative) factor analysis, comprising the generative model  (Sec.<ref>) and the variational inference network (Sec.<ref>).
It further elaborates on variational inference (Sec.<ref>) and discusses techniques for uncertainty evaluation (Sec. <ref>). 


 §.§ Bayesian generative model



In classical FA, an observation matrix   X 
is factorized into two latent variables X ∼(θ Φ), where both θ and Φ are drawn from distributions such as Gaussian.
While FA is intuitive for feature learning, it may not be effective for learning meaningful representations with its reconstruction training objects <cit.>. 
Motivated by the connection between CL and MF, as indicated by Eq. <ref>, we build contrastive factor analysis (CFA) by instead modeling the co-occurrence matrix A̅, which represents the relationship among positive and negative samples. 



Unlike classical FA, which focuses solely on modeling the generative process between the latent variable θ and target data X, CFA requires modeling the generative process among observation data samples X, the latent variable θ, and the corresponding normalized co-occurrence matrix A̅, as depicted in Figure. <ref>.
To address this, we draw inspiration from conditional VAE <cit.> to devise a modified FA. 
Specifically, given data samples and augmentations X ∈ℝ^N × D and the corresponding normalized co-occurrence matrix A̅∈ℝ+^N × N, CFA factorize A̅∈ℝ+^N × N under the Gaussian likelihood as follows:

    A̅_i,j x_i,x_j∼𝒩(  θ_i θ_j), θ_i x_i∼𝒩( f(x_i),   1 ), θ_j x_j∼𝒩( f(x_j),   1 ).

where θ_i,  θ_j  ∈ℝ_+^d is the factor score
matrix, each column of which encodes the relative importance of each atom in a sample. 

Further, recent studies suggest that modeling latent features using non-negative vectors can harness the advantageous properties of non-negative matrix factorization (MF), which facilitates the acquisition of  disentangled representations <cit.>.
Hence, we draw inspiration from relevant research <cit.> to develop a contrastive non-negative FA (CNFA) approach.
Specifically, the CNFA factorizes A̅ into sparse and non-negative gamma latent variables, which can be described as follows:

    A̅_i,j x_i, x_j∼𝒩(  θ_i θ_j), θ_i x_i∼( f(x_i), 1 ), θ_j x_j∼( f(x_j), 1 ).


In our formulation, the ideal scenario involves modulating the prior of the latent variables θ_i by the input x_j. However, we can readily relax this constraint to ensure that the latent variables are statistically independent of the input variables <cit.>. Thus, for contrastive factor analysis (CFA), we simply define the data-independent prior as f( x_i) = 0, while for contrastive non-negative factor analysis (CNFA), we set f(x_i) = 1.









 §.§ Reparameterizable latent variable distributions






  
Variational Gaussian posterior for CFA. 
The Gaussian distribution θ∼𝒩(μ, σ^2) has probability density function f( θμ ,σ ^2) = 1/√(2σ ^2π)e^ - ( x - μ)^2/2σ ^2, where θ∈ℝ. It is reparmeterizable as drawing 
θ∼𝒩(μ, σ^2) us equivalent to letting θ = g(ε) := μ + εσ ,   ε∼𝒩(0,1). The KL divergence is analysis as ( 𝒩( μ _1,σ _1^2) || 𝒩( μ _2,σ _2^2)) = logσ _2/σ _1 + σ _1^2 + ( μ _1 - μ _2)^2/2σ _2^2.
As shown in Fig. <ref>, the variational inference network utilize the obtained latent features h_i from a deep feature extractor, such as ResNet <cit.>, to 
construct the variational posterior:

    q(θ_i  x_i ) = 
    	𝒩(μ_i, σ^2_i ), 
    	μ_i = f_μ(h_i),
    	σ^2_i = (f_σ( h_j)),
where f_·^(l)(·) denotes the neural network, and  applies (1+(·)) non-linearity to each element to ensure positive Gaussian variance parameters. 
It's evident that the CFA will reduce to matrix factorization 
as σ of the Gaussian distribution goes to zero. Therefore, the proposed CFA can be viewed as a generalization of
vanilla contrastive learning <cit.>.



  
Variational Weibull posterior for CNFA: 
While the gamma distribution is ideal for the posterior due to its encouragement of sparsity and satisfaction of non-negativity constraints, its non-reparameterizability and incompatibility with gradient descent optimization lead to the adoption of the Weibull distribution as an approximation for the posterior of gamma latent variables.

The reason for choosing the Weibull distribution is threefold <cit.>:  ) the Weibull distribution is
similar to a gamma distribution, capable of modeling sparse,
skewed and positive distributions;
) the Weibull distribution has a simple
reparameterization so that it is easier to optimize; That is, to sample θ∼( k,λ) with probability density function p( θ k,λ) = k/λ ^kθ ^k - 1e^ - ( S/λ)^k, where θ∈ℝ_+, it is equivalent to letting 
θ = g(ε) := λ( - ln (1 - ε ))^1/k,  ε∼(0,1).
) The KL divergence from the gamma to Weibull distributions has an analytic expression as
( (k,λ)  || (α,β) )  = γα/k - αλ+ 
k + βλΓ(1+1/k) - γ -1 -α(β) + Γ(α).
where γ is the Euler-Mascheroni constant and  Γ is the gamma function. This provides an efficient way to estimate the training
objective, which we will discuss in detail in the next subsection.
Similarly, the variational inference network employs the obtained latent features _i to 
construct the variational posterior:










    0.95q(θ_i | x_i ) = Weibull(k_i, λ_i ), 
        k_i = Softplus(f_k(h_i)), 
        λ_i = ReLU(f_λ(h_i))/Γ(1 + 1 /k_i )


where f_·(·) denotes the neural network, and  applies (1+(·)) non-linearity to each element to ensure positive Weibull shape and scale parameters. 
With above inference network, there are 𝔼( θ) = ( f_λ( h_i)).
It's evident that the CNFA will reduce to non-negative matrix factorization if the parameter k of the Weibull distribution goes to infinity.
Therefore, the proposed CNFA can be viewed as a generalization of
vanilla non-negative contrastive learning <cit.>.




 §.§ Variational inference


For the contrastive factor analysis, given the model parameters referred to as W, which consist of the parameters in the generative model and inference network, marginal likelihood is defined as:

    0.95p ( A̅W)  = ∫∏_i = 1^N ∏_j = 1^N p( A̅_i,j  |  θ_i, θ_j)∏_j = 1^N p( θ _j   |   x _j) d θ _j = 1^N .

The inference task is
to learn the parameters of the generative model and the inference network.
Similar to VAEs,
the optimization objective of contrastive factor analysis can be achieved by
maximizing the evidence lower bound (ELBO) of log-likelihood as

    0.95ℒ(A̅)   =  ∑_i = 1^N ∑_j = 1^N 𝔼_q(θ_j  x_j)[ ln p(A̅_i,j  |   θ _i, θ _j)]  -  ∑_j = 1^N 𝔼_q(θ_j  x_j)[ lnq(θ _j  |   x_j)/p(θ _j) ] .
 
where the first term is the expected log-likelihood of the generative model, which ensures reconstruction performance, and the second term is the Kullback–Leibler (KL) divergence that constrains the variational distribution q(θ_j) to be close to its prior p(θ_j).
Unfortunately, Equation. <ref> becomes intractable with factor analysis algorithms as the input space X grows exponentially large with N →∞. To address this challenge, we reformulate the factor analysis problem (Equation. <ref>) equivalently as a tractable contrastive learning (CL) objective that only necessitates sampling positive and negative samples from the joint and marginal distributions, respectively:

    0.95ℒ = 
        - 2𝔼_x,  x + (θ_+^T  θ_+) 
     + 𝔼_x,   x_+( θ_-^T θ_-)^2
       -  𝔼_q(θ_+ x_+)[ lnq(θ _+  |   x_+)/p(θ _+) ] 
    -  𝔼_q(θ_- x_-)[ lnq(θ _-  |   x_-)/p(θ _-) ]
 
 The parameters in contrastive (non-negative) factor analysis can be directly optimized by advanced gradient algorithms, like Adam <cit.>.



  
Complexity analysis 
 Our framework is computationally and memory-efficient because only a few parameters need to be added to infer the distribution parameters. 
 The extra memory cost comes from the posterior inference network, which only adds d × 1 parameters due to σ_i^2 ∈ℝ^1 and k_i∈ℝ^1, where d is the dimension of feature h_i.
 Meanwhile, the additional computations involve the sampling process and computing the KL term, which is of scale O(d). 
 Both of above
 is inconsiderable compared to the memory and computational scale of the deep backbone neural network <cit.>. 



 §.§ Uncertainty-aware representation
 
To quantify the inferred latent variable uncertainty, we adopt Shannon's entropy <cit.> of the variational posterior. Specifically, The Shannon's entropy of Gaussian variational posterior with parameters (μ, σ) is defined as: 

    0.95H ( q_ϕ( θ _ix_i)) = ∑_m = 1^d ln( σ _m√(2π e))
. And Shannon's entropy of the Weibull variational posterior with parameters (λ, k) is defined as: 

    0.95H( q_ϕ( θ _ix_i)) = ∑_m = 1^d ( k_m - 1)γ/k_m + lnλ _m/k_m + 1

where γ stands for Euler's constant. 
It should be noted that the higher the entropy, the higher the uncertainty.



§ RELATED WORK

Our work is mainly related to factor analysis and probabilistic representation learning. Here, we introduce the related research.

Factor analysis. 
Before the era of deep learning, factor analysis was widely used in representation learning. And its non-negative extension <cit.> has achieved great progress in text analysis. Further, there is increasing interest in hierarchical factor analysis, which is aimed at exploring hierarchical data representation. 
But all this work is different from ours,  which mainly focuses on improving factor analysis with contrastive learning. 
Besides, <cit.> proves contrastive learning is capable of revealing latent variable posterior information in FA. 
Nonetheless, our approach diverges from this earlier work as it places more emphasis on proposing a novel model that can effectively learn representations.

Probabilistic representation learning. 
Probabilistic representation learning has indeed made significant strides in recent years and has garnered considerable attention in the machine learning community. 
Its ability to capture uncertainty and model complex data distributions has proven immensely useful in various applications such as natural language processing <cit.>, computer vision <cit.>, and graph processing <cit.>.
In contrast to the previous efforts, this paper develops a general framework for (non-negative)  contrastive factor analysis that benefits from both (non-negative) factor analysis and contrastive learning. 
Besides, there are efforts to improve contrastive learning with the Bayesian method, which is aimed at evaluating the uncertainty <cit.>. 
Unfortunately, most of these techniques rely on a decoder, which frequently degrades the performance of downstream tasks <cit.>. 
On the opposite hand, our research is an encoder-only approach that can enhance downstream tasks' performance. 


§ EXPERIMENT



 §.§ Feature learning



  
Baselines & Datasets. 
To evaluate the expressiveness of the proposed model, we compare it with strong baselines, including SimCLR(CL) <cit.> and non-negative contrastive learning(NCL) <cit.>. 
More specifically, we compare CFA with CL and CNF with NCL.
It should be noted that CFA is built on SimCLR codebase [<https://github.com/vturrisi/solo-learn>] with a minor modification to the hidden projector output.
We evaluate linear probing and fine-tuning performance on three benchmark datasets: CIFAR-10, CIFAR-100<cit.>, and Imagenet-100<cit.>. 
Apart from in-distribution evaluation, we also compare models on three out-of-distribution datasets, including stylized ImageNet<cit.>, ImageNet-Sketch<cit.>, ImageNet-C<cit.>(restricted to ImageNet-100 classes). 
For all experiments, we run 3 random trials and report their mean and standard deviation. 
All experiments are performed on
workstation equipped with a CPU i7-10700 and accelerated
by four GPU NVIDIA RTX 4090 with 24GB VRAM. 




  
Experiment Setting.
In our experiments, Resnet-18<cit.> is used as the backbone, and the model is trained for 200 epochs on CIFAR-10, CIFAR-100, and 100 epochs for ImageNet-100. 






  
In distribution experiment results. The experiment results are  exhibited in Table <ref>. 
Firstly, compared to the FA model, CFA has achieved a significant improvement, primarily attributed to the effective contrastive learning training strategies and corresponding expressive deep neural networks.
Secondly, acting as the Bayesian extension of contrastive learning (CL) and non-negative contrastive learning (NCL), respectively, CFA and CNFA demonstrate varied degrees of improvement across different datasets. This result indicates the efficacy of stochastic latent variables in capturing complex data dependencies and enhancing overall performance <cit.>.
Finally, when compared with CFA, CNFA consistently exhibits superior performance across all datasets, as NCL performs better than CL. This improvement may stem from the ability of non-negative stochastic latent variables to learn  disentangled representations <cit.>.



Out-of-distribution experiment results. Apart from the expressiveness evaluation, we further evaluate the model's robustness on the out-of-distribution dataset, and the results are depicted in Table <ref>. 
Firstly, compared to deterministic contrastive learning methods, CFA and CNFA show significant improvement in out-of-distribution generalization downstream tasks.
The improved OOD performance can be partially attributed to the generalization introduced by the stochastic latent variables, which allows the model to handle unseen and diverse data better <cit.>.
By inheriting the advantages of non-negative factor analysis, CNFA can learn disentangled representation <cit.> and exhibit better transfer ability compared with CFA.





 §.§ Uncertainty evaluation

In addition to their enhanced expressiveness and robustness on out-of-distribution (OOD) data, another notable strength of CFA and CNFA is their ability to capture data uncertainty.


Evaluation Metric. 
 With the entropy of the testing result and a given entropy threshold, we can determine whether the model is certain or uncertain about one prediction.  
 To evaluate the
uncertainty estimates, we use Patch Accuracy vs Patch Uncertainty (PAvPU) <cit.>, which is
defined as PAvPU = (n_ac + n_iu)/(n_ac + n_au + n_ic + n_iu), where n_ac, n_au, n_ic, n_iu  represent the numbers of accurate and certain, accurate and uncertain, inaccurate and certain, and inaccurate and uncertain samples, respectively. Compared to Top-1 Accuracy= (n_ac + n_au)/(n_ac + n_au + n_ic + n_iu),
 PAvPU metric would be higher if the model tends to
generate accurate predictions with low uncertainty and inaccurate predictions with high uncertainty leading to a more trustworthy algorithm.


Evaluation Setting.
Firstly, utilizing Equation <ref> and Equation <ref>, we calculate the information entropy of each sample in the test set. Subsequently, we select the M samples with the highest information entropy as uncertain samples, while the remaining samples are considered definite. In practice, we conducted experiments with different values of M, specifically selecting 100, 200, and 400 samples as uncertain samples to compute the test indices.





Experiment results:
Table. <ref>  show the PAvPU results of CFA and CNFA on ImageNet-100. CFA and CNFA achieves better performance on PAvPU metrics compared to Top-1 accuracy. 
This suggests that CFA and CNFA are effective not only in making correct predictions but also in identifying when it is likely to be wrong, which is crucial for applications requiring high reliability. 

 


Qualitative analysis:
To further validate the efficacy of the proposed model in uncertain evaluation, we conducted qualitative analysis. Specifically, we selected 20 images with the highest and lowest uncertainty predicted by CNFA from the ImageNet-100 dataset, and the experimental results are illustrated in Figure <ref>.
Figure. <ref> visually indicates that images with noisy backgrounds exhibit higher levels of uncertainty. 
In contrast, Figure.  <ref> shows that a clean background alongside high contrast is prone to displaying low uncertainty. 
This clear distinction highlights the effectiveness of CNFA in identifying the factors contributing to uncertainty in image predictions. 

Ablation study:
To further investigate the relationship between prediction uncertainty and downstream performance, we conducted an ablation study. Specifically, we sorted the samples in the test set based on their evaluation uncertainty and divided the set into five subsets accordingly. Subsequently, we calculated the average accuracy and corresponding average uncertainty for each subset. We then plotted histograms depicting the average uncertainty and accuracy for these five subsets, as illustrated in Figure. <ref>. The histogram clearly indicates a negative correlation between uncertainty and accuracy: higher uncertainty correlates with lower accuracy. It's worth noting that these findings are supported by qualitative analysis, where higher uncertainty suggests noisier and more challenging-to-classify data.










































































































    


















 


 §.§ Disentanglement representation learning

Learning a robust representation involves extracting explanatory factors that are sparse, disentangled, and semantically meaningful <cit.>. By incorporating non-negative constraints, NCL can learn disentangled features <cit.>. On the other hand, the Gamma distribution promotes sparsity due to its probability density function, which helps filter out irrelevant features and focus on the most significant ones. Thus, we evaluate CFA's performance in disentangled representation learning.


  
Evaluation Metric. 
Many disentanglement metrics like MIG <cit.> are supervised and require ground-truth factors which are often unavailable in practice. Consequently, current disentanglement evaluation is often limited to synthetic data<cit.>. To validate the disentanglement on real-world data, we adopt an unsupervised disentanglement metric SEPIN@k<cit.>. SEPIN@k measures how each feature f_i(x) is disentangled from others f_≠ i(x) by computing their conditional mutual information with the top k features, i.e., SEPIN@k= 1/k∑_i=1^kI(x, f_r_i(x)|f_≠ r_i(x)), which are estimated with InfoNCE lower bound <cit.> implemented following <cit.>.



  
Experiment Result. Table <ref> demonstrates that CNFA features show better disentanglement than CL and NCL in all top-k dimensions. The advantage is larger by considering the top features, since learned features may contain noisy dimensions <cit.>. Incorporating proper prior distribution (e.g., Gamma distribution) will effectively mitigate this problem. Thanks to the inherent properties of the Gamma distribution, CNFA tends to produce disentangled features, making it more effective in distinguishing the underlying factors of variation within the data. In conclusion, CNFA achieves better feature disentanglement on real-world data and exhibits exceptional performance in estimating uncertainty. 






§ CONCLUSION

In conclusion, this paper introduces a novel framework called Contrastive Factor Analysis (CFA), which bridges the gap between traditional Factor Analysis (FA) and the powerful technique of contrastive learning. By leveraging the mathematical equivalence between contrastive learning and matrix factorization, CFA capitalizes on the strengths of both methods, offering enhanced expressiveness, robustness, and interpretability.
Further, by extending CFA to a non-negative version (CNFA), we exploit the interpretability properties of non-negative factor analysis, enabling the learning of disentangled representations. Through extensive experimental validation, our methodology demonstrates efficacy across various dimensions, including expressiveness, robustness, interpretability, and accurate uncertainty estimation.
In essence, CFA and CNFA represent promising approaches that amalgamate the insights from factor analysis and contrastive learning, paving the way for more effective unsupervised representational learning in the deep learning era.




unsrtnat










































 Paper Title: Leaf Angle Estimation using Mask R-CNN and LETR Vision Transformer.







Leaf Angle Estimation using Mask R-CNN and LETR Vision Transformer


    Venkat Margapuri
Department of Computing Sciences 

Villanova University

Villanova, PA, USA 


Prapti Thapaliya
Department of Computing Sciences 

Villanova University

Villanova, PA, USA 


Trevor Rife
Pee Dee Research and Education Center  

Clemson University

Florence, SC, USA 



    August 5, 2024
==================================================================================================================================================================================================================================================================================================



§ ABSTRACT 
Modern day studies show a high degree of correlation between high yielding crop varieties and plants with upright leaf angles. It is observed that plants with upright leaf angles intercept more light than those without upright leaf angles, leading to a higher rate of photosynthesis. Plant scientists and breeders benefit from tools that can directly measure plant parameters in the field; i.e., on-site phenotyping. The estimation of leaf angles by manual means in a field setting is tedious and cumbersome. We mitigate the tedium using a combination of the Mask R-CNN instance segmentation neural network, and Line Segment Transformer (LETR), a vision transformer. The proposed Computer Vision (CV) pipeline is applied on two image datasets, Summer2015-AmesULA and Summer2015-AmesMLA, with a combined total of 1,827 plant images collected in the field using FieldBook, an Android application aimed at on-site phenotyping. The leaf angles estimated by the proposed pipeline on the image datasets are compared to two independent manual measurements using ImageJ, a Java-based image processing program developed at the National Institutes of Health and the Laboratory for Optical and Computational Instrumentation. The results, when compared for similarity using the Cosine Similarity measure, exhibit ≈0.98 similarity scores on both independent measurements of Summer2015-AmesULA and Summer2015-AmesMLA image datasets, demonstrating the feasibility of the proposed pipeline for on-site measurement of leaf angles.



Fieldbook, ImageJ, Leaf Angle Estimation, Line Transformer (LETR), On-site Phenotyping, Mask R-CNN




§ INTRODUCTION

The yields and planting densities of maize in the United States have increased concurrently during the past 50 years <cit.>. A comparative analysis of U.S. commercial maize hybrids released since the 1960s revealed that by selecting high yielding hybrids under high planting densities, breeders indirectly selected hybrids with upright leaf angles (LAs) <cit.>. The discovery was that upright LAs combined with higher planting densities improve light distribution within the canopy. For instance, modern hybrids intercept 14% more light than older hybrids <cit.>. Leaf Angle Distribution (LAD) is a key parameter that describes the structure of horizontally homogeneous vegetation canopies. It is defined as the probability of a leaf element of unit size to have its normal within a specified unit solid angle <cit.>. LAD affects the manner in which incident photosynthetically active radiation is distributed on plant leaves, thus directly affecting plant productivity. The traditional technique to measure LAD involves the use of mechanical inclinometer, a precision instrument that measures the angle of slope of an object with respect to its gravity by creating an artificial horizon. The use of the mechanical inclinometer makes the process of determining LAD laborious and time-consuming. Other techniques such as 3-D digitizing of individual plant elements using specialized instrumentation <cit.> and laser scanning <cit.> are available. However, these techniques are resource-demanding. A feasible, and inexpensive alternative is the use of digital imagery to estimate LA since image analysis preserves the state of the canopy while providing enough insight into the morphometry. Image processing tools such as ImageJ <cit.> and Adobe Photoshop have built-in angle estimation tools that aid in the estimation of LA. The pitfall is that the operation of the tools is manual which means that the estimations may be subject to user bias and users may spend extended amounts of time to estimate LAs, in case of large datasets. In order to overcome the pitfalls of the manual setup, this paper proposes an automated technique for LA estimation using Mask R-CNN <cit.> instance segmentation neural network, and Line Segment Transformer (LETR) <cit.>, a vision transformer. 



§ RELATED WORK

Dzievit <cit.> conducted genetic mapping and a meta-analysis to dissect genetic factors controlling LA variation on maize. Genetic mapping populations were developed using inbred lines B73 (Iowa Still Stalk Synthetic), PHW30 (lodent, expired plant variety protection inbred), and Mo17 (Non-Stiff Stalk) that have distinct LA architectures. The leaf angles were estimated using the ImageJ image processing tool.   Herbert <cit.> described a technique for the estimation of leaf angles using stereo-photogrammetry. The technique permitted accurate measurement of leaf angle and position from several meters away and had sufficient resolution to permit the analysis of complex phenomena such as the effect of leaf shape upon interception of light and photosynthesis.   Sinoquet et al. <cit.> proposed a method to measure light interception by vegetation canopies using a 3-D digitizer and image processing software. The 3-D digitizer allowed for simultaneous acquisition of the spatial coordinates of leaf locations and orientations. The software for image synthesis also had the ability to make virtual photographs of the real canopy. The information on light interception was derived from virtual images by using the simple features of image analysis software.   Hasoi et al. <cit.> estimated the LAD of wheat canopy at different growth stages such as tillering, stem elongation, flowering, and ripening stages by using a high-resolution portable scanning lidar. The canopy was scanned three-dimensionally by optimally inclined laser beams emitted from several measuring points surrounding the canopy and 3-D point cloud images in each stage were obtained. After co-registration of lidar images between different measurement positions, leaves were extracted from the images and each leaf was divided into small pieces along the leaf-length direction. Each of the pieces was approximated as a plane, to which normal (perpendicular) was estimated. The distribution of the leaf inclination angles was derived from the angles of the normal with respect to the zenith.   Ryu et al. <cit.> examined the feasibility of seven techniques such as litterfall, allometry, LAI-2000, TRAC, digital hemispheric photography, digital cover photography, and traversing radiometer system to determine leaf area index across a 9-ha domain in an oak-savanna ecosystem in California, USA. It was shown that the combination of digital cover photography and LAI-2000 could provide spatially representative leaf area index, gap fraction and element clumping index.



§ MATERIALS AND METHODS

Researchers at Iowa State University captured the images of several plants in a field using the Fieldbook android application <cit.>. As for the current work, two datasets namely, Summer2015-AmesULA and Summer2015-AmesMLA wherein the former consists of 872 images and latter, 955 images, are used as the experimental datasets. Each of the images in either dataset shows the intersection of the leaf and stem. A sample image is as shown in Fig. <ref>.



The proposed leaf angle estimation procedure contains two broad steps: 1.	Extraction of region of interest (ROI) 2.	Estimation of Leaf Angle



 §.§ Extraction of Region of Interest (ROI)

The first step in the extraction of ROI is the detection of leaf and stem within the image. The images captured in the field contain numerous plants that look similar. Hence, it is important to determine the plant of interest and perform a curated extraction of it for the estimation of LA. Conventionally, a feasible technique for foreground extraction is OpenCV’s GrabCut <cit.> algorithm. However, the results obtained from the application of the GrabCut algorithm on each of the datasets are not satisfactory. While the algorithm captures the foreground, it also captures part of the plants in the background. The captured plants in the background are essentially noise and detrimental to LA estimation. Fig. <ref> shows the botched foreground extraction performed on a plant image from the Summer2015-AmesMLA dataset.



In order to better perform foreground extraction, the CNN model of Mask R-CNN <cit.> <cit.> is employed to extract foreground and identify the region of interest appropriately. Mask R-CNN is a deep neural network aimed to solve the problem of instance segmentation in computer vision. Instance Segmentation <cit.> is the task of precisely identifying the pixels of each of the objects in the image. It is perhaps the hardest of the computer vision (CV) tasks of classification <cit.>, semantic segmentation <cit.>, object detection <cit.>, and instance segmentation.


    
  * 175 images belonging to each of the Summer2015-AmesMLA and Summer 2015-AmesULA datasets are manually annotated using makesense.ai tool. The tool is free to use and generates annotations in the COCO format that Mask R-CNN is able to consume. A polygonal annotation is drawn where the stem and leaf meet each other. Fig. <ref> shows a polygonal annotation drawn using makesense.ai on an image with leaf and stem.

    


  * The annotated images are trained on the Mask R-CNN network for a total of 20 epochs. The loss of the network after 20 annotations is 0.034. However, the loss is not significant since Mask R-CNN is used to extract the portion of the image where the leaf and stem meet, but not to extract masks that are precise in shape and size. All the annotated images are used for training only. The dataset is not split into training, validation, and test datasets since the goal is not to evaluate the performance of Mask R-CNN, but to extract regions of interest in the image that are later processed to estimate the angle between leaf and stem. Upon training, the trained weights are used to extract the masks of each of the images in the Summer2015-AmesMLA and Summer2015-AmesULA datasets. Fig. <ref> shows the mask extracted by Mask R-CNN for the plant image shown in Fig. <ref>.




  * The predicted mask indicates the region of interest that corresponds to the leaf and stem. In order to extract the region from the original image, the 'bitwise AND' operation is applied on the mask predicted by Mask R-CNN and original image. The resulting image gives the region of interest in the original image. Fig. <ref> shows the region of interest extracted from the original image. Please note that the image is zoomed in for clarity.


    






 §.§ Estimation of Leaf Angle

The basis for the estimation of leaf angle is provided by <cit.>. The work describes the above-ear and below-ear leaf angles that are measured by plant scientists. Fig. <ref>, an excerpt from <cit.>, shows the exact position at which the angle between leaf and stem is to be measured. While leaves are typically long and tilt in different directions owing to environmental factors, the angle between leaf and stem is measured exactly where the leaf and stem conjoin.   The estimation of leaf angle is performed using Line Segment Transformer (LETR), a vision transformer <cit.>. Derived from transformers in the world of Natural Language Processing (NLP), a Vision Transformer (ViT) is a type of neural network that is used to perform CV tasks, such as image classification, and object detection. The basic principle of the transformer used for NLP is the representation of words as sequences. Drawing inspiration from it, ViT works by representing an image as a sequence of patches. A patch is a small rectangular size of the image, typically 16x16 pixels. Each patch that is extracted from the image is represented as a feature vector. The feature vectors are extracted using a convolutional neural network (CNN), such as ResNet-50 or VGG-16. The extracted feature vectors are input to an encoder network. The encoder network is a stack of self-attention <cit.> layers, where self-attention is a mechanism that allows the model to identify and establish long-range dependencies between the patches of images extracted from the original image. The encoder network outputs a sequence of vectors that may be fed into a multi-layer perceptron (MLP) for CV tasks such as classification, and detection.   The LETR vision transformer is an adaptation of the Detection Transformer (DETR). The DETR is an object detection model developed by Facebook (Meta) that leverages the transformer architecture. The ResNet architecture is typically preferred as the feature extractor within DETR to extract features from patches of an image. The DETR model follows the encoder-decoder architecture similar to the conventional transformer, with the use of multi head self attention being the key difference between the two. Multi head self attention is the idea where the Attention module comprises N (multiple) attention heads and each attention head repeats its computations multiple times in parallel. All the attention computations are combined to yield a final attention score. Bipartite Matching computed using the Hungarian Algorithm <cit.> is used to compute the difference between the ground truth bounding boxes and predicted bounding boxes of each of the objects in the image. The LETR vision transformer follows the DETR model, but is designed specifically for the identification of line segments. The LETR model improves upon the DETR model's ability to detect geometric structures by augmenting the loss term. 




    
  * Xu et al. trained the LETR model on two datasets namely, Wireframe <cit.> and YorkUrban <cit.> using the feature extractors of ResNet-101 and ResNet-50 respectively. The pretrained weights of the LETR model trained on the Wireframe dataset are used for the experiment.
    
  * The images from the Summer2015-AmesULA and Summer2015-AmesMLA datasets are evaluated using the pretrained weights. The line segments output by the model are plotted on evaluation images, as shown in Fig. <ref>.

    
    
  * However, it is the case that several of the detected line segments indicate the stem rather than the leaf, as shown in Fig. <ref>. The lines on the stem are irrelevant and must be ignored for the estimation of leaf angle. Two constraints, one on the slope and another on proximity to the image boundary are placed on the detected line segments to filter out the ones that don’t belong to the leaf.
    
        
  * Slope Constraint: The leaf is generally oriented in comparison to the stem that is almost perpendicular to the ground, in the majority of cases. As a result, line segments whose orientations are between 80 and 90 degrees are ignored, since they are likely to belong to the stem but not the leaf.
        
  * Boundary Proximity Constraint: It is empirically determined that the line segments which belong to the leaf are at least 100 pixels apart from the image boundary. The line segments that are closest to the image boundary are those of the stem. Hence, the constraint to ignore line segments that are within 100 pixels of the image boundary works to ignore line segments that belong to the stem. Both constraints are required to be satisfied for a line to be ignored. In certain images, the orientation of the leaf is between 80 and 90 degrees. In such cases, without the boundary constraint, the lines pertinent to the leaf are ignored. The line segments that pass the filter are assumed to be present on the leaf.
    
    
  * Compute the slope of each of the line segments and find the mode of the slopes. Mode refers to the most frequently occurring item among a list of items.
    
  * Retrieve the line segments with the most frequently occurring slope for further processing. Only the line segments with the orientation of the mode are considered to eliminate any stray lines that may be detected on the leaf. In the absence of mode i.e. no two line segments have the same slope, the line segment whose slope is the median of all slopes is considered.
    
  * Compute the orientation of the line segment made with the horizontal as the tan inverse of slope i.e., angle = tan^-1(slope). The orientation made by the line segment is deemed the orientation of the leaf with the stem.
    





§ IMPACTING FACTORS




 §.§ Impact of Image Sharpness

Image Sharpness describes the clarity of detail in a photo. The aspects of resolution and acutance primarily impact the sharpness of an image. It is imperative that the images be sharp for the line segments to be identified precisely by LETR. Multiple failures are observed with the detection of line segments in images of low sharpness. However, when the sharpness is increased, the line segments are well detected.



 §.§ Multiple Leaf and Stem Detections

In a few cases, the trained Mask R-CNN model detects multiple instances on a given image, some not pertinent to the leaf and stem. Fig. <ref> shows two images where multiple instances are detected. In such cases, the detection of the greatest size (contour) is considered the instance of interest, i.e. leaf and stem. However, it may be flawed in cases where the identified contour is larger than the actual leaf and stem instance in the image. 





§ RESULTS


The algorithm is applied to each of the images from the Summer2015-AmesULA and Summer2015-AmesMLA datasets, and the orientation between leaf and stem is estimated. The results are compared with the leaf angle measurements made by two graduate students at Iowa State University. The students worked independently of each other as they arrived at their estimates using the ImageJ tool. The measurements by the students and the proposed algorithm are compared using a metric known as Cosine Similarity <cit.>. Cosine Similarity is the cosine of the angle between two vectors that are typically non-zero and within an inner product space.  It is useful to compare the similarity between two vectors represented in a higher-dimensional vector space. Mathematically, it is defined as the division between the dot product of vectors and the product of the magnitude of each vector and is expressed as, similarity = A · B/|A||B| where A and B are the two vectors compared for similarity. The measure is expressed as a value between 0 and 1.   On the Summer2015-AmesMLA dataset, the cosine similarity between the proposed algorithm and student 1 is 0.98 which indicates a ten-degree difference in orientation between vectors, and the proposed algorithm and student 2 is 0.99 which indicates an eight-degree difference in orientation between the vectors. On the Summer2015-AmesULA dataset, the cosine similarity between the proposed algorithm and student 1 is 0.998 which indicates a four-degree difference in orientation between vectors, and the proposed algorithm and student 2 is 0.992 which indicates a seven-degree difference in orientation between the vectors.   Furthermore, an intuitive outlier analysis is performed on the results to gain insight into the similarity of estimation between the different techniques. The rule used to identify an outlier is, “the difference in estimates between the proposed algorithm and a student is greater than eight degrees”. The analysis assumes that the measurements made by the students are accurate. However, the estimations by the students also have a variance between them. The variance in student estimates, given by Σ(e1-e2)/n where e1 and e2 are estimates made by students 1 and 2 respectively, and n is the total size of the data sample, are 2.57 and 2.07 on Summer2015-AmesMLA dataset and Summer2015-AmesULA dataset respectively. Hence, the eight-degree range to identify outliers is reasonable. Applying the outlier rule, a total of 31 outliers on the Summer2015-AmesMLA dataset and 16 outliers on the Summer2015-AmesULA dataset are detected. Among the estimates that are not outliers, a variance of 3.73 degrees and 3.11 degrees is identified between the proposed algorithm and student 1, and the proposed algorithm and student 2 respectively on the Summer2015-AmesMLA dataset whereas a variance of 2.67 degrees and 2.72 degrees is identified between the proposed algorithm and student 1, and proposed algorithm and student 2 respectively on the Summer2015AmesULA dataset. The results show that the proposed algorithm outputs results that highly co-relate with the manual measurements, demonstrating the feasibility of the technique for leaf angle estimation.



§ FUTURE WORK AND CONCLUSION

Moving forward, the proposed algorithmic pipeline will be implemented as part of a real-time Android application so plant scientists and individual enthusiasts may leverage the application on a day-to-day basis. The bottleneck for the mobile application is the use of neural network models. The current Mask R-CNN network will have to be adapted so it works on mobile devices using networks such as Mobilenet SSD v1/v2 <cit.> as the backbone in place of the current ResNet-101 model. The annotation of images is another bottleneck that is laborious and time-consuming. The use of a screen to act as the common background over the plants as images are captured is an efficient way to help identify the location of the plant in the image.   Overall, the proposed technique demonstrates the idea of estimating the angle between leaf and stem using instance segmentation neural networks and vision transformers, a task that is laborious and cumbersome when performed manually. We will implement the leaf angle estimation as a feature in the FieldBook <cit.> Android application. The estimation of leaf angles using automation is novel and has the potential to turn into a one-of-a-kind utility for the agricultural community. 

00
Dzievit
M Dzievit and X Li and J Yu,
Dissection of leaf angle variation in maize through genetic mapping and meta‐analysis,
The plant genome,
12,
1

Zou
X Zou and M Mottus and P Tammeorg and C L Torres and T Takala and J Pisek and P Pellikka,
Photographic measurement of leaf angles in field crops,
Agricultural and Forest Meteorology,
184,
137-146

Sinoquet
H Sinoquet and S Thanisawanyangkura and H Mabrouk and P Kasemsap,
Characterization of the light environment in canopies using 3D digitising and image processing,
Annals of Botany,
 82,
2,
203 - 212

Hasoi 
F Hasoi and Y Nakai and K Omasa,
Estimating the leaf inclination angle distribution of the wheat canopy using a portable scanning lidar,
Journal of Agricultural Meteorology,
65,
3,
297-302

Herbert 
T J Herbert,
Stereophotogrammetry of plant leaf angles,
Photogrammetric Engineering and Remote Sensing,
61,
1,
89-92

Ryu 
Y Ryu and O Sonnentag and T Nilson and R Vargas and H Kobayashi and R Wenk and D D Baldocchi,
How to quantify tree leaf area index in an open savanna ecosystem: a multi-instrument and multi-model approach,
Agricultural and Forest Meteorology,
150,
1,
63 - 76

Kiryati
N Kiryati and Y Eldar and A M Bruckstein,
A probabilistic hough transform,
Pattern Recognition,
24,
4,
303 - 316

He
K He and G Gkioxari and P Dollar and R Girshick,
Mask r-cnn,
Proceedings of the IEEE conference on Computer Vision,
2017,
2961 - 2969

Bharati
P Bharati and A Pramanik,
Deep learning techniques-R-CNN to mask R-CNN,
Computational Intelligence in Pattern Recognition,
657 - 668,
2020

Tian
D Tian and Y Han and B Wang and T Guan and H Gu and W Wei,
2021,
Review of object instance segmentation based on deep learning,
Journal of Electronic Imaging,
31,
4

Lu
D Lu and Q Weng,
A survey of image classification methods and techniques for improving classification performance,
International journal of Remote Sensing,
28,
5,
823 - 870


Lateef
F Lateef and Y Ruichek,
2019,
Survey on semantic segmentation using deep learning techniques,
Neurocomputing,
338,
321 - 348

Liu
L Liu and W Ouyang and X Wang and P Fieguth and J Chen and X Liu and M Pietikainen,
2020,
Deep learning for generic object detection: A survey,
International journal of computer vision,
128,
2,
261 - 318

Schneider
C A Schneider and W S Rasband and K W Eliceiri,
NIH Image to ImageJ: 25 years of image analysis,
2012,
Nature Methods,
9,
7, 
671 - 675

Xu
Xu, Yifan and Xu, Weijian and Cheung, David and Tu, Zhuowen,
Line segment detection using transformers without edges,
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition,
4257–4266,
2021

Ramachandran
Ramachandran, Prajit and Parmar, Niki and Vaswani, Ashish and Bello, Irwan and Levskaya, Anselm and Shlens, Jon,
Stand-alone self-attention in vision models,
Advances in neural information processing systems,
32,
2019

Sahbani
Sahbani, Bima and Adiprawita, Widyawardana,
Kalman filter and iterative-hungarian algorithm implementation for low complexity point tracking as part of fast multiple object tracking system,
2016 6th international conference on system engineering and technology (ICSET),
109–115,
2016,
IEEE

Huang
Huang, Kun and Wang, Yifan and Zhou, Zihan and Ding, Tianjiao and Gao, Shenghua and Ma, Yi,
Learning to parse wireframes in images of man-made environments,
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition,
626–635,
2018

Denis
Denis, Patrick and Elder, James H and Estrada, Francisco J,
Efficient edge-based methods for estimating manhattan frames in urban imagery,
Computer Vision–ECCV 2008: 10th European Conference on Computer Vision, Marseille, France, October 12-18, 2008, Proceedings, Part II 10,
197–210,
2008,
Springer

Talbot
J F Talbot and X Xu,
2006,
Implementing grabcut,
Brigham Young University

Vijaymeena
M K Vijaymeena and K Kavitha,
2016,
A survey on similarity measures in text mining,
Machine Learning and Applications: An International Journal,
3,
2,
19 - 28

Pedregosa
F Pedregosa and G Varoquaux and A Gramfort and V Michel and B Thirion and O Grisel and E Duchesnay,
2011,
Scikit-learn: Machine learning in Python,
The journal of machine learning research,
12,
2825 - 2830

Chiu
Y C Chiu and C Y Tsai and M D Ruan and G Y Shen and T T Lee,
Mobilenet-SSDv2: An improved object detection model for embedded systems.,
2020,
International conference on system science and engineering (ICSSE),
1 - 5

Rife
T Rife and J Poland,
Crop Science,
Fieldbook: an open-source application for field data collection on android,
54,
4,
2014,
1624 - 1627





 Paper Title: LoopSparseGS: Loop Based Sparse-View Friendly Gaussian Splatting.



[
	
        

	Revocable Backdoor for Deep Model Trading
    [
    Accepted for publication in Astrophysics & Space Sciences (ApSS)
====================================================================


    < g r a p h i c s >

	figureIn scenarios with limited input data, the standard 3D Gaussian Splatting (3DGS) method generates insufficient points and minimal depth constraints for training. Our LoopSparseGS employs additional pseudo-cameras to produce more comprehensive initialization points and richer depth information for 3DGS training. Additionally, we found that the excessively large ellipsoids, damage view rendering quality through blurring. To mitigate the issue, we propose a Sparse Friendly Sampling (SFS) strategy to split oversized ellipsoids. The results are presented in the third column, demonstrating the effectiveness of our method.
        

	
]
Revocable Backdoor for Deep Model Trading
    [
    Accepted for publication in Astrophysics & Space Sciences (ApSS)
====================================================================


[1] Github: https://github.com/pcl3dv/LoopSparseGShttps://github.com/pcl3dv/LoopSparseGS
[2] Corresponding authors

§ ABSTRACT 

Despite the photorealistic novel view synthesis (NVS) performance achieved by the original 3D Gaussian splatting (3DGS), its rendering quality significantly degrades with sparse input views. This performance drop is mainly caused by the limited number of initial points generated from the sparse input, insufficient supervision during the training process, and inadequate regularization of the oversized Gaussian ellipsoids. To handle these issues, we propose the LoopSparseGS, a loop-based 3DGS framework for the sparse novel view synthesis task. In specific, we propose a loop-based Progressive Gaussian Initialization (PGI) strategy that could iteratively densify the initialized point cloud using the rendered pseudo images during the training process. Then, the sparse and reliable depth from the Structure from Motion, and the window-based dense monocular depth are leveraged to provide precise geometric supervision via the proposed Depth-alignment Regularization (DAR). Additionally, we introduce a novel Sparse-friendly Sampling (SFS) strategy to handle oversized Gaussian ellipsoids leading to large pixel errors. Comprehensive experiments on four datasets demonstrate that LoopSparseGS outperforms existing state-of-the-art methods for sparse-input novel view synthesis, across indoor, outdoor, and object-level scenes with various image resolutions. 




§ INTRODUCTION



Novel view synthesis (NVS) aims to generate photorealistic images of 3D scenes from perspectives that were not originally captured <cit.>, which is an essential task in computer vision and graphics field. Recently, 3D Gaussian Splatting (3DGS) <cit.> has emerged as a promising technique for NVS, as it can efficiently model the highly detailed appearance and geometry of 3D scenes.
Such superior performance is usually obtained when large amounts of input images are available. 

However, in many real-world applications <cit.>, only a few sparse input images are available such as in sports event broadcasting and robotics, where acquiring dense views is often time-consuming and expensive, even impossible. 
These sparse inputs introduce several challenges to 3DGS. 
Firstly, given the sparse input views, the initial Gaussian points provided by Structure from Motion (SfM) <cit.> can be sparse and inadequate, as shown in Fig. <ref> (top left). Secondly, reconstructing the appearance and geometry of scenes becomes an under-constrained and ill-posed issue with insufficient inputs with only the image reconstruction constraints. 

Thirdly, the scales of some Gaussians grow to be very large during the optimization process, and these oversized Gaussian ellipsoids result in the overfitting problem, thus producing unsatisfactory results at novel viewpoints as illustrated in Fig. 1 (top middle).. 




Recent studies <cit.> have attempted to address the aforementioned issues of sparse-input 3DGS. 
To handle the issue of sparse initialized Gaussian points, FSGS <cit.> introduces a proximity-guided Gaussian unpooling technique, which generates new Gaussians by measuring the proximity of existing Gaussians with their neighbours during training. This densification strategy, however, is sensitive to noisy points, potentially leading to the generation of invalid points. 
To mitigate the under-constrained problem, DNGaussian <cit.> adopts monocular depth, obtained from the pre-trained monocular depth estimator <cit.>, to constrain the depth rendered by 3DGS in a global-local normalization manner. However, these monocular depth maps are often scale-inconsistent across different views, posing a challenge for effective depth regularization. 
Additionally, previous studies ignore the issue of excessively large Gaussians in sparse-input scenarios, limiting the quality of novel view synthesis.


In this paper, we present the LoopSparseGS, a novel 3DGS framework for precise and robust sparse-input novel view synthesis. LoopSparseGS is built upon a looping mechanism and incorporates a sparse-friendly Gaussian densification strategy with the following considerations. 



As shown in Fig. <ref> (bottom left), we observe that those rendered views close to the training views exhibit high visual quality even with sparse input. 
This observation motivates us to integrate these pseudo images, i.e., rendered novel images, with training images to generate additional initialized 3D points using SfM. 
This process is developed in a looping mechanism to increase the number of initialized points. 
Consequently, we propose a Progressive Gaussian Initialization (PGI) strategy, which leverages both rendered images and training images to iteratively increase initialized Gaussian points, resulting in more comprehensive scene coverage. 

Moreover, increased initialized 3D points provide additional precise but sparse depth constraints for 3DGS optimization. Dense monocular depth from the pre-trained model provides dense but scale-invariant depth constraints. To effectively utilize the two constraints, we develop a Depth-alignment Regularization (DAR) approach to generate smoother and more precise rendered depths as illustrated in (d) of Fig. <ref>. 






Furthermore, to address the issue of excessively large Gaussian ellipsoids, we propose a Sparse-friendly Sampling (SFS) strategy guided by the pixel error. Specifically, SFS identifies and splits the Gaussian ellipsoids associated with high-error pixels that have the largest weights, which could
effectively produce more detailed geometric and rendering results, as shown in Fig. <ref> (bottom middle). 
The main contributions of this work are as follows: 

    
  * We present the LoopSparseGS, a novel 3DGS-based framework for sparse-input novel view synthesis, featuring a looping mechanism to provide denser Gaussian initialization and precise geometry constraints, and a sparse-friendly sampling strategy to address the oversized Gaussian ellipsoids. 
    
  * We develop a Progressive Gaussian Initialization (PGI) method to produce dense 3D Gaussian points by incorporating iteratively rendered images with training images into SfM.
    
  * We propose a Depth Alignment Regularization (DAR) approach that aligns dense relative-scale monocular depths with absolute-scale sparse SfM-derived depths, to provide effective geometric constraints.
    
  * We introduce a Sparse-friendly Sampling (SFS) strategy to address the issue of excessively large Gaussian ellipsoids unique to sparse-input scenes, thus further enhancing the view synthesis quality of the scene. 
    
    
    
    
  * Comprehensive experimental results on four datasets demonstrate that our proposed approach outperforms existing state-of-the-art methods in novel view synthesis with sparse-input data, across indoor scenes, outdoor scenes, and object-level scenes.    



§ RELATED WORK



 §.§ Novel View Synthesis using Radiance Fields
 
Novel view synthesis techniques typically utilize one or more input views to generate images from novel perspectives. Recent advancements in this field have concentrated on employing radiance fields and achieved encouraging progress.
For example, Mildenhall et al. <cit.> introduce Neural Radiance Field (NeRF) that enables novel view synthesis using coordinate-based neural networks. 
Tremendous following efforts concentrate on improving its rendering quality <cit.>, efficiency <cit.>, scene understanding <cit.>, and 3D generation <cit.>. 
Particularly, Mip-NeRF <cit.> employs conical frustum instead of single rays to reduce aliasing. Mip-NeRF 360 <cit.> extends this approach to handle unbounded scenes.  
Recently, Kerbl et al.<cit.> achieve a significant breakthrough with the 3D Gaussian Splatting (3DGS) method, which enhances rendering efficiency using explicit Gaussian representations and the differentiable rasterization technique. 
Building upon its high efficiency in novel view synthesis, several works attempt to extend 3DGS to various tasks. Wu et al. <cit.> propose an explicit representation method for dynamic scenes utilizing 3D Gaussian and 4D neural voxels. Tang et al. <cit.> presents a generative 3D Gaussian Splatting model for efficient text-to-3D content creation. Zhou et al. <cit.> introduce DrivingGaussian for efficient dynamic autonomous driving scene reconstruction. 

Although the methods mentioned above demonstrate excellent performance in novel view synthesis, they typically require dense input views for training the radiance fields. When provided with sparse training views, these methods tend to overfit the available training views, resulting in a significant performance drop in novel views. 




 §.§ Sparse Novel View Synthesis




In recent years, several NeRF-based studies have been proposed to address sparse-input novel view synthesis. 
Specifically, RegNeRF <cit.> introduces geometry and color regularization from unobserved viewpoints, enhancing the quality of sparse-input novel view synthesis. It employs a 2D consistency loss on the depth and color of image patches, ensuring that neighboring regions have similar geometry and appearance. 
InfoNeRF <cit.> enhances sparse-input view synthesis by employing regularization techniques based on information theory. Specifically, it applies a sparsity constraint on the density distribution of the ray by minimizing entropy.
DS-NeRF <cit.> utilizes sparse depth cues generated by SfM, to impose depth supervision for sparse NeRF.  
ViP-NeRF <cit.> enhances the traditional NeRF framework by incorporating the visibility prior, which enforces multi-view constraints during optimization. This modification involves calculating the visibility of a point and using these results to regularize the visibility and alpha-blended depth across different views. 
FreeNeRF <cit.> incorporates a frequency regularization strategy designed to train the sparse-input NeRF, aiming to regularize the frequency range of NeRF’s inputs, and the other to penalize the near-camera density fields. 
SparseNeRF <cit.> utilizes a pre-trained depth estimation model to generate pseudo-ground truth depth maps, which are employed for a local depth ranking loss. Besides, SparseNeRF applies a depth smoothness loss to ensure that the rendered depth maps exhibit patch-wise smoothness.

In addition, some 3DGS-based approaches try to tackle the sparse-input novel view synthesis. 
For example, Yu et al. <cit.> align sparse depth from SfM with dense depth from a monocular depth estimation model <cit.> to guide the geometry for sparse-input 3DGS. 
FSGS <cit.> integrates estimated monocular depth and employs a Pearson correlation depth distribution loss to train sparse 3D Gaussian Splatting. 
Likewise, DNGaussian <cit.> introduces a monocular depth loss and incorporates global-local depth normalization to optimize the parameters of Gaussians. 
SparseGS <cit.> proposes generative constraints from a pre-trained diffusion model <cit.>, which guides the 3D Gaussian representation in novel views via Score Distillation Sampling. 


Unlike previous methods that utilize scale-inconsistent monocular depth across different views for regularization, our work introduces a Depth-alignment Regularization (DAR) approach. DAR extract accurate and reliable depth values from the SfM points and aligns them with monocular depths using a sliding-window mechanism, providing more effective geometric supervision. 
Furthermore, our work proposes a loop-based Gaussian initialization, resulting in a denser point cloud. This not only offers more precise depth values for the DAR but also facilitates the training convergence quality of Gaussians. 
Additionally, we present a sparse-friendly sampling strategy to further enhance Gaussian densification.   




















§ METHOD


LoopSparseGS facilitates scene novel view synthesis given sparse input images, and the framework is illustrated in Fig. <ref>. 
First, initialized sparse point clouds and camera parameters are obtained using Structure from Motion (SfM). We then introduce a loop-based initialization strategy using pseudo-view rendering results to progressively provide denser Gaussian initialization (Section <ref>). 
Second, during the Gaussian optimization, we incorporate depth-alignment regularization to impose additional and precise geometric constraints (Section <ref>). 
Lastly, we adopt a sparse-friendly Gaussian densification approach to sample effective Gaussians for sparse-input reconstruction quality enhancement (Section <ref>). 
Before introducing our method, we briefly revisit 3D Gaussian Splatting in Section <ref>. 





 §.§ Preliminaries
  
3D Gaussian Splatting (3DGS) <cit.> represents a 3D scene using a set of anisotropic 3D Gaussian primitives, enabling efficient and differentiable rendering via α-blending.   
The properties of i-th Gaussian primitive can be described as Θ_i = { u_i, o_i, s_i, q_i, c_i }, where u_i∈ℝ^3 is the center, o_i∈ℝ is the opacity, s_i ∈ℝ^3 is the scaling factor, q_i ∈ℝ^4 is the rotation, and c_i ∈ℝ^3 is the color.  
To compute the pixel color C, 3DGS employs differentiable α-blending point-based rendering by blending 𝒩 Gaussian points in the front-to-back depth order, which can be written as: 

    C = ∑_i ∈𝒩𝐜_i α_i ∏_j=1^i-1 (1 - α_j),
 
where 𝒩 denotes the set of Gaussian points that overlap with the given pixel, and α_i is calculated by α_i = o_i f_i^2D, where f_i^2D represents the projection function of the i-th Gaussian onto the 2D plane. 

3DGS is optimized by projecting 3D Gaussians onto the 2D image plane and employing gradient-based color supervision to minimize the distance between the rendered image Ĩ and the ground truth image I. This process is as follows: 

    ℒ_color = (1-λ)ℒ_1(Ĩ, I) + λℒ_D-SSIM(Ĩ, I),

where λ is set to 0.2 as per <cit.>. 





 §.§ Progressive Gaussian Initialization (PGI)
  


Initialization of Gaussian points is crucial for 3DGS-based novel view synthesis, as it significantly impacts the training convergence quality and speed. The original 3DGS relies on dense input images to generate enough initial Gaussian points. However, in scenarios with limited views, the number of 3D points drops dramatically, potentially compromising reconstruction quality. 
Considering that rendered views close to the training views exhibit satisfactory visual quality, as illustrated in Fig. <ref> (bottom left), we develop a Progressive Gaussian Initialization (PGI) approach, which combines the rendered images with the original training images to generate additional initialized points. Instead of generating images once, we rely on the iteratively refined 3DGS to produce the high-quality pseudo images progressively. The detailed process is shown in Fig.<ref>. 




Before starting a new loop,  we generate 4 new pseudo-views around each training view, resulting in a total of P × 4 new pseudo-views per loop iteration, where P denotes the number of training views. 
The camera location of the pseudo-views is obtained by adding Gaussian noise to the training camera positions. 
The locations of the generated pseudo views are computed as follows:


    z_gl^ij = N(z_tj ,  ε +δ× l)^i, j∈ [1,P], i∈ [1,4]

Where z_g and z_t are locations of the pseudo views and training views, respectively. N is Gaussian noise, j denotes the j-th training view, and i denotes the i-th pseudo-view generated from one of the training views for each loop. 
To ensure the quality of the generated pseudo-view, ε starts from a small value. It gradually enlarges by a rate of δ with loop iterating (l). 
ε and δ are set to 0.02 and 0.1, respectively. 
As the number of loops increases, the pseudo view gradually expands the coverage around the training views.
To force the view range of the pseudo-images under the coverage of the training views, the locations of pseudo-images should not be out of the bounding box determined by the locations of the training views. The orientation of pseudo-images is the averages of two adjacent training views as in <cit.>.
For each loop, pseudo-images generated from previous loops are accumulated to train the Gaussians. 









 §.§ Depth-alignment Regularization (DAR)
  
















Beyond providing denser initialized Gaussian points, our loop-based method offers an additional advantage: valuable and reliable depth information derived from these initialized 3D points. 
Given the inherent inaccuracies and sparsity in the depth supervision from these 3D points, we propose a Depth-alignment Regularization (DAR) strategy, which comprises a Filter Enhancement and a Sliding window-based Alignment.
The former aims to enhance the accuracy of SfM-derived depth while the latter incorporates dense monocular depth cues to improve depth regularization for sparse-input 3D Gaussian optimization. 



Filter Enhancement.
We observed that inaccurate depth information directly derived from SfM can result in erroneous rendering, as illustrated in Fig. <ref>. The horns and leaves scenes exhibit significant holes at their edges. To enhance depth reliability, we implemented three filtering strategies according to the reliability and visibility of matched points from certain perspectives. 

Filter Strategy 1: Firstly, we ignore the depth of 3D points with large match errors according to the SfM key point match report. We use the average pixel match error to find coarse points. The threshold is set to 2. This filter strategy can be described with the following equation.


    D(p)=
            d, if Υ (p)<2,
            
    
            0, if Υ (p)>2,

where Υ is the average pixel match error and d is the depth value. Note that such points are not used for depth map generation but are yet kept for Gaussian initialization. Although such coarse points can not produce precise depth information they are accurate enough to initialize Gaussian.

Filter Strategy 2: Secondly, considering that pseudo-view images may coincidentally have erroneous regions that satisfy the key points match, we discard points generated solely from pseudo-view. This is illustrated as follows:

    p=
          Keep,  if p ⊂ 
                M(C^t_i,C^t_j) || M(C^t_i,C^g_j),
                
    Discard,  if p ⊂    
                M(C^g_i,C^g_j),

where p is the matched 3D point. C^t and C^g are the training views and pseudo views, respectively. The M indicates that the point is computed from these views. Points are unreliable when they are produced from pseudo-images only rendered by the preliminary trained 3DGS.






Filter Strategy 3: Lastly, considering that the foreground points block the background points in 3D space, we select 3D points to produce the depth images based on visibility provided by the RGB images.
For certain view C_i, its corresponding depth D_i of certain 3D point p_j is d, only if p_j derived from view C_i. This can be described under the following conditions:

    D_i(p_j)=
                d, if p_j⊂ M(C_i,C_k),
                
    
                0, other situations,

where D_i(p_j) represents that the depth of point p_j in i-th camera. C_i is the i-th camera. d represents the distance from point p_j to camera C_i.
















Sliding window-based Alignment.
To impose dense depth regularization, an intuitive approach is to employ the image-level Pearson distribution loss between the rendered depth and the monocular depth predicted by the MiDaS model <cit.>. 
However, directly combining SfM-derived sparse depth and dense monocular depth constraints leads to a misalignment issue, manifesting as erroneous black points in the rendered depth maps, as illustrated in (c) of Fig. <ref>. 



This phenomenon is attributed to the limitations of the relative-scale and image-level Pearson constraints, which struggle to impose sufficient constraints on local regions, thereby hindering the alignment of absolute-scale and sparse depths derived from SfM.

To address this misalignment issue, we develop a sliding window-based depth regularization, as illustrated in Fig. <ref>. 
We slide the window from left to right, and from the top to down with over the monocular depth and render depth. We compute the Pearson distribution loss of the window-covered region rather than across the entire image. This region-based Person distribution loss can effectively work with the L_1 loss generated from sparsely distributed depth values from SfM as they both are effective locally.
Therefore, the final loss function (L_a) is formulated as follows: 

    L_a = λ_dL_1(D_p, D_c) + ∑_w=1^Wλ_p L_p[χ_w (D_p), χ_w (D_m)],

where D_p, D_c, and D_m denote the rendered depth, SfM-derived depth, and mono-depth, respectively. χ_w represents the w-th window, and W is the total number of windows.



This local window-based depth-alignment constraint approach not only effectively enforces the absolute scale depth constraint, but also allows the depth within the window to satisfy the relative scale distribution, thus providing depth-aligned depth constrains for more precise rendered depth as shown in (d) of Fig. <ref>.
















































 §.§ Sparse-friendly Sampling (SFS)
  





The Gaussian ellipsoid densification scheme in 3DGS performs effectively with an large number of training images. However, with extremely sparse input data, some Gaussian ellipsoids may grow excessively large with extremely sparse input data and lead to inferior rendered results, as illustrated in (a) of Fig. <ref>. This is caused by two primary factors: 
1) The initial scale or size of Gaussian ellipsoids is determined by the average distance from its three nearest neighbours. In cases where the sparse initialized point cloud fully covers the entire space, this method can lead to irrationally large scales for some ellipsoids. 
2) The extreme sparsity of input data can cause rapid increases in some ellipsoids' scale along certain directions. This leads to overfitting the training views while significantly deteriorating the performance of novel viewing perspective. 




One obvious solution is to increase the frequency of Gaussian densification or lower the Gaussian densification threshold. However, these direct strategies would exacerbate the overfitting of Gaussian Splatting in sparse input settings. To address it,
we introduce a non-trivial strategy, named Sparse-friendly Sampling, which selectively applies densification to Gaussian ellipsoids that adversely affect rendering. In this way, we enhance the representation of Gaussian ellipsoids of large PSNR error area by splitting the oversized Gaussian ellipsoids to produce more Gaussian ellipsoids. 




 Figure. <ref> (a) shows that the oversized ellipsoids could lead to the pixels blurring, resulting in lower PSNR values. Based on such observations, we traverse all rendered pixels of the training views and collect pixels with the highest PSNR errors. Since the color of each pixel is derived from multiple ellipsoid primitives through α-blending, we identify the ellipsoid with the largest weight ω (calculated by ω = α_i ∏_j=1^i-1 (1 - α_j)) as the primary determinant of the pixel's final color. Subsequently, we apply a splitting procedure to the Gaussian primitives, akin to the method employed in the original 3D Gaussian Splatting (3DGS). This process subdivides large ellipsoids into m smaller ellipsoids to enhance the representation of fine details (in our experiments, m = 2).
Besides, we introduce opacity regularization during the training process to encourage non-maximum weight ellipsoids along a camera ray to be more transparent. These operations can reduce the number of excessively large ellipsoids and enhance detailed geometric and rendered results as shown in (b) of Fig. <ref>.





















 §.§ Optimization
 
We summarize our training constraints as follows:

    ℒ =  λ_1L_1(C_p,C_gt)+λ_2L_D-SSIM(C_p,C_gt)+L_a+L_o,

where C_p and C_gt denote the rendered and GT images, respectively. L_1 and L_D-SSIM represent the photometric and SSIM loss. L_a is the depth-alignment loss computed as Eq. <ref>. L_o is the non-maximum weight regularization: 

    L_o=λ_o/N∑_n=1^N |α_n   |,

where N denotes the total number of non-maximum weighted ellipses hit by all pixels in an image.

In addition, we compute L_1, L_D-SSIM and L_o for training views and L_a for both training views and pseudo views.




§ EXPERIMENT









 §.§ Experimental Settings

Datasets. 
To evaluate our sparse-input method, we conduct experiments on four widely used sparse-view datasets for novel view synthesis: LLFF <cit.>, DTU <cit.>, Mip-NeRF360 <cit.>, and Blender <cit.>. 
The LLFF dataset <cit.> includes eight forward-facing scenes. 
Following previous methods <cit.>, we select every eighth image as the held-out testing view, and evenly sample sparse views from the remaining images as the training views. 
For each scene, three views are utilized to train all the approaches. 
During evaluation, the image resolution are set to 1008 × 756 and 504 × 378. 

The Mip-NeRF360 dataset <cit.> consists of nine scenes, each containing a complex central area or object against an intricate background. 
As per the protocol in <cit.>, we utilize seven of these scenes for our experiments, employing 24 views with images downsampling rates of 4 and 8 for training all methods. 

The DTU dataset <cit.> is a comprehensive object-level dataset. In line with <cit.>, we use the same 15 scenes with three views for training in the experiments. 
Consistent with the evaluation protocol of prior research <cit.>, object masks are employed to exclude the background during inference, as evaluating the entire image introduces bias due to background elements. 

The Blender dataset <cit.> includes eight objects rendered with photorealistic images using Blender. Following <cit.>, we use 8 images for training and 25 unseen images for testing.




















Evaluation Metrics. 
For quantitative comparisons, we adopt three evaluation metrics: peak signal-to-noise ratio (PSNR), structural similarity index (SSIM) <cit.>, learned perceptual image patch similarity (LPIPS) to assess the visual quality<cit.>. 



Implementation Details.

Our approach is implemented using the PyTorch framework, and utilizes the pre-trained Midas Model <cit.> for zero-shot monocular depth estimation and the Colmap model for initial camera poses and 3D points. We set the total number of loop iterations as 3 and the number of pseudo cameras generated around one training camera l as 4. In a single loop iteration, we start the densification of Gaussian ellipsoids after 1000 iterations, and the frequency of densification is set to 200 iterations. The 2D grad threshold of densification is set to 0.0005.

We set the window length to 32 and the step size of the window sliding to 4 for all datasets.
For each loop, 3DGS are trained with 10,000 iterations. The weights of the loss function λ_1, λ_2, λ_o, λ_d, λ_p are set to 0.8, 0.2, 0.05, 0.005 and 0.05, respectively.

Efficiency. 
Our method achieves average rendering speeds (ARS) of 418, 490, 831, and 720 FPS for the LLFF (3 training views), Mip-NeRF360 (24 training views), DTU (3 training views), and Blender (8 training views) datasets, respectively. 

Here, the average speed is calculated by  ARS=1/20∑_i=1^20M/T, where T is the total inference time of all test images across all scenes in one dataset, the M is the number of all test images, and we tested 20 times to eliminate random error.


The training time of the model depends on the number of loops, with a training time of about 10 min per loop and a total training time of about 35-45 min on all datasets.
All experiment times were evaluated on a 3090 NVIDIA GPU.








 §.§ Comparison With Existing Methods

Comparisons on LLFF.
As shown in Table <ref>, our proposed method outperforms other state-of-the-art approaches across different image resolutions and sparse-input settings. 
Specifically, our approach surpasses the second-best FSGS by 0.42 and 0.38 in PSNR at two test resolutions when only using 3 sparse-input views for training. 

Moreover, we can see that more training views can bring better reconstruction quality and our LoopSparseGS delivers superior performance compared to all other methods, validating the effectiveness of our proposed strategies. 
We show the qualitative results in Fig. <ref>. 
Existing methods tend to produce artifact and blurry rendered results. In comparison, our approach exhibits fine-grained details such as the leaves (Scens: Flower) and the floor (Scens: Room). 







Comparisons on Mip-NeRF360.
Table <ref> presents the quantitative results in complex scenes from Mip-NeRF360. 
It can be seen that our method also outperforms other state-of-the-art approaches in terms of various metrics across different image resolutions. 
Compared to Mip-NeRF requiring dense-input, although methods using regularizations or depth information for sparse-input, such as FreeNeRF and SparseNeRF, enhance rendering quality to some extent, they still encounter a performance bottleneck. 
Compared to FSGS that incorporates Gaussian unpooling densification technique and monocular depth maps, our proposed method significantly outperforms it with an improvement of 0.39 and 1.02 in PSNR across two resolutions. These demonstrate the effectiveness of our proposed loop-based mechanism and importance-guided sampling strategy.   
Moreover, we provide qualitative comparison in Fig. <ref>. It can be seen that existing methods tend to produce blurred rendered results with incomplete structure, as highlighted by the red boxes around the “plate," “switch," and “window." 
In contrast, our proposed loop-based and sparse-friendly sampling strategies yield denser initialized Gaussians with effective geometric constraints, resulting in more complete structures and finer details. 








Comparisons on DTU.
As shown in Table <ref>, we present the quantitative results on the DTU 3-view sparse-input setting. For the object-level scenes, our method also achieve the best rendering quality in terms of different metrics, with the significant improvement of 0.76 in PSNR. 
In Fig. <ref>, we present the visual comparisons. 
Compared to other methods that produce blurry renderings, our approach can capture color details much closer to the ground truth, demonstrating its effectiveness on object-level real-world scenes. 


Comparisons on Blender.
Table <ref> shows the quantitative results on the Blender dataset with an 8-view sparse-input setting. 
Our method also significantly outperforms other approaches on the synthesis scenes, achieving a 0.92 improvement in PSNR compared to the second-best method. 
Furthermore, Fig. <ref> illustrates the rendered results. 
DNGaussian struggles to achieve precise texture and illumination, whereas our approach accurately captures the geometry of objects and authentic reflections. This demonstrates the effectiveness of our proposed strategies for object-level synthesis scenes. 


 























 §.§ Ablation Experiments

In this section, we ablate our method on the LLFF 3-view with 503 × 381 image resolution setting. The quantitative results are presented in Tables <ref> to 
<ref>.



  §.§.§ Effectiveness of architecture modules

In Table <ref>, we present the ablation results obtained by progressively applying our Depth-alignment Regularization (DAR), Progressive Gaussian Initialization (PGI), and Sparse-friendly Sampling (SFS) strategies. 
Compared to the baseline (a), each proposed module contributes to improved reconstruction quality, as shown in (b)-(d). Specifically, 
DAR provides additional depth information to mitigate the under-constrained effect caused by sparse-input, thereby enhancing reconstruction quality. 
PGI introduces a loop-based initialization strategy and offers denser Gaussians for scene modeling, resulting in a 0.803 increase in PSNR. 
SFS provides a Gaussian densification strategy suitable for sparse-view training and exhibits a gain of 0.282 PSNR. 
Furthermore, as shown in (e)-(g), the combination of these strategies yields better results, with all three working together to produce the best performance, as demonstrated in (h). 






  §.§.§ Effectiveness of depth-alignment loss

Table <ref> presents the ablation of our depth-alignment loss through various depth loss configurations. 
The experiments indicate that both SfM depth loss and monocular depth loss enhance the sparse-input reconstruction quality, with their combination yielding even better results. 
Moreover, applying our depth-alignment regularization with window-level Pearson correlation loss can further boost the performance. 
This improvement is attributed to that our window-level operation can effectively align the absolute depth and relative depth constraints and rectify incorrect depth constraints. 













  §.§.§ Effectiveness of filter strategies

As shown in Table <ref>, we present the effectiveness of the proposed filter strategies through various configurations. 
It can be seen that each filter strategy contributes to improved reconstruction quality, as shown in (b)-(d). 
Moreover, their combination can achieve better results, demonstrating their effectiveness in eliminating unreliable geometric constraints and providing informative supervision for sparse-input view synthesis. 




  §.§.§ Effectiveness of sparse-friendly sampling

In Table <ref>, we investigate the effectiveness of the non-max weight regularization and max weight densification operations in SFS. 
The experiments show that both operations enhance the representation of Gaussian ellipsoids, leading to improved PSNR. 





  §.§.§ Number of looping

To investigate the number of looping in the Progressive Gaussian Initialization (PGI) strategy, we present comparison results in Table <ref>. 
Increasing the number of loops results in more initialization points, which can enhance rendering performance. However, excessive looping may lead to performance degradation. This is primarily because pseudo-views added at later stages may be distant from the training views, leading to inaccurate initialization points and ultimately impairing performance. Therefore, we selected three loops as the final setting for our experiments. 





  §.§.§ Number of pseudo views

To examine the impact of the number of pseudo views in the Progressive Gaussian Initialization (PGI) strategy, we conducted ablation experiments, as shown in Table <ref>. 
The results indicate that adding 12 pseudo images at each Gaussian initialization, equivalent to four times the training views produces better results. Thus, we adopt this as the final configuration. 






















§ CONCLUDING REMARKS



This paper proposes LoopSparseGS, an innovative 3DGS-based framework for novel view synthesis using sparse input data.  
In LoopSparseGS, the proposed loop-based strategies, including Progressive Gaussian Initialization (PGI) and Depth-alignment Regularization (DAR), provide denser Gaussians and precise geometric information for effective scene coverage. 
Additionally, the proposed Sparse-friendly Sampling (SFS) strategy enhances Gaussian densification unique to sparse-input scenes, facilitating the generation of photo-realistic images.
Extensive experimental results demonstrate that our approach outperforms existing SOTA methods in sparse-input novel view synthesis across indoor, outdoor, and object-level scenes at various image resolutions. 






    ieeenat_fullname
    







 Paper Title: DistillGrasp: Integrating Features Correlation with Knowledge Distillation for Depth Completion of Transparent Objects.


	
	
	
	IEEE Robotics and Automation Letters. Preprint Version. Accepted Month, Year
	FirstAuthorSurname et al.: ShortTitle 
	
	
	
	
	
	
	
	DistillGrasp: Integrating Features Correlation with Knowledge Distillation for Depth Completion of Transparent Objects
    Yiheng Huang^0009-0007-9527-6569,
			Junhong Chen^0000-0002-4874-9550, 
			Nick Michiels^0000-0002-7047-5867, 
			Muhammad Asim^0000-0002-6423-9809Y. Huang, J. Chen, M. Asim and W. Liu are with the College of Computer Science and Technology, Guangdong University of Technology, Guangzhou, 510006, China. E-mails: (huangyiheng.gdut@gmail.com, CSChenjunhong@hotmail.com, asimpk@gdut.edu.cn, liuwy@gdut.edu.cn.)  (Yiheng Huang and Junhong Chen contributed equally to this work.)    (Corresponding author: Junhong Chen and Wenyin Liu.),  
			Luc Claesen^0000-0003-0405-6290, and Wenyin Liu^0000-0002-6237-6607
				J. Chen and N. Michiels are with Hasselt University - Flanders Make, Expertise Centre for Digital Media, Hasselt, Belgium.
				L. Claesen is with Hasselt University, Hasselt, Belgium.
			 W. Liu is also with Zhongguancun Laboratory, Beijing, China.
		
	
    
===================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================

empty  
	empty 
	
	
	§ ABSTRACT 
		Due to the visual properties of reflection and refraction, RGB-D cameras cannot accurately capture the depth of transparent objects, leading to incomplete depth maps. To fill in the missing points, recent studies tend to explore new visual features and design complex networks to reconstruct the depth, however, these approaches tremendously increase computation, and the correlation of different visual features remains a problem. To this end, we propose an efficient depth completion network named DistillGrasp which distillates knowledge from the teacher branch to the student branch. Specifically, in the teacher branch, we design a position correlation block (PCB) that leverages RGB images as the query and key to search for the corresponding values, guiding the model to establish correct correspondence between two features and transfer it to the transparent areas. For the student branch, we propose a consistent feature correlation module (CFCM) that retains the reliable regions of RGB images and depth maps respectively according to the consistency and adopts a CNN to capture the pairwise relationship for depth completion. To avoid the student branch only learning regional features from the teacher branch, we devise a distillation loss that not only considers the distance loss but also the object structure and edge information. Extensive experiments conducted on the ClearGrasp dataset manifest that our teacher network outperforms state-of-the-art methods in terms of accuracy and generalization, and the student network achieves competitive results with a higher speed of 48 FPS. In addition, the significant improvement in a real-world robotic grasping system illustrates the effectiveness and robustness of our proposed system.
	
	
	
		Distillation learning, depth completion, transparent object grasping.
	
	
	
	
	

§ INTRODUCTION

	Transparent objects are widely used in various scenes, such as manufacturing, household services, etc. However, the perception of transparent objects has been recognized as a critical challenge <cit.> due to their reflective and refractive surfaces that cannot be perceived by the depth cameras, resulting in inaccurate and missing depth values. Without accurate depth maps, many downstream robotics applications cannot be applied, e.g., object grasping, autonomous driving, human-robot interaction, etc.
	
	The task of the depth completion of transparent objects is to recover the complete depth maps from the input of RGB images and raw depth maps, which contains two aspects: correcting the drifting depth values caused by refraction and generating the missing depth values caused by reflection. To prevent the drifting depth values from affecting the learning of the network, authors in <cit.> adopted a Deeplabv3+ network to remove the depth of transparent objects, and then extracted multiple visual maps as complementary information to complete the depth maps. However, the extraction of various visual maps not only increases the computational cost but also brings the problem of feature correlation. Instead of leveraging multiple visual maps, Chen et al. <cit.> introduced Transformer to the depth completion of transparent objects in order to acquire contextual information for completing depth maps. Hong et al. <cit.> integrated the DenseNet and Swin Transformer to extract both local and global features from RGB images and concatenated them with the depth maps to a U-Net architecture network for depth completion. Although these methods utilize advanced networks for fine-grained feature extraction, they still suffer from huge computational costs due to the complexity of the networks.
	
	In order to exploit the advantages of advanced networks while also maintaining model efficiency, in this paper, we propose a novel approach called DistillGrasp which preserves the advanced networks as the teacher networks to transfer the knowledge to the low-capacity student networks. Specifically, for the teacher network, we devise a position correlation block (PCB) which uses a transformer-based method to capture the structure information and establish the correspondence between RGB images and depth maps. For the student network, an efficient consistent feature correlation module (CFCM) is designed to capture the pairwise relationship based on reliable regions according to their consistency. To ensure that the student network can learn comprehensive features, a distillation loss incorporating distance, object structure, and edge information is introduced. Extensive experiments conducted on the ClearGrasp dataset verify the effectiveness and robustness of our methods. 
	
	In summary, our main contributions are as follows:
	
		
  * To the best of our knowledge, this is the first work on depth completion of transparent objects using knowledge distillation
		
		
  * For the teacher and student network, we separately devise two different correlation strategies named position correlation block (PCB) and consistent feature correlation module (CFCM) to capture the pairwise relationship, which guarantees accuracy and speed.
		
		
  * Considering the gap between the teacher and student network, we design a composite distillation loss consisting of distance loss, structural loss, and edge loss to ensure the student network can learn both local and global features.
		
		
  * Extensive experiments conducted on the ClearGrasp dataset illustrate that our teacher network outperforms state-of-the-art methods in accuracy and generalization, while the student network achieves competitive results with higher speed. The successful deployment of our system on a UR10e robot for grasping transparent objects verifies the effectiveness and robustness of our method.
	
	
	
	

§ RELATED WORK

	

 §.§ Depth Completion for Transparent Object Perception

	Depth completion for transparent objects aims to generate the missing depth values by referring to the existing RGB images and depth maps. Existing methods can be divided into two categories based on the number of viewpoints <cit.>, the multi-view approaches and single-view approaches. The multi-view approaches complete the depth maps of transparent objects by utilizing information collected from different perspectives. For example, Klank et al. <cit.> proposed to match two different views to locate the contradictory places, and then leveraged triangulation to correct the wrong point clouds. Authors in <cit.> introduced the Neural Radiance Field (NeRF) to depth completion for transparent objects, but they suffered from huge computation costs. Compared with multi-view approaches, single-view approaches not only avoid capturing multiple views, but also reduce the time of processing large amounts of data from different perspectives. The primary single-view approaches are designed based on particular patterns <cit.>, which cannot handle diverse scenarios and objects. To extend the application scenes, Sajjan et al. <cit.> proposed an end-to-end network to extract multiple visual maps from a large amount of data to complete the missing depth. Hong et al. <cit.> developed a U-Net network to complete the depth based on the correct depth filtered from the reflective and refractive regions. However, their methods focused on local features and ignored contextual information. Authors in <cit.> extended the Transformer to depth completion for transparent objects, offering an enhanced perception of global information. Although the utilization of Transformer can improve accuracy, it also significantly increases computational costs, hindering the deployment in real-world environments.
	
	

 §.§ Knowledge Distillation in Robotic Application

	Knowledge distillation <cit.> was first proposed to transfer refined knowledge from sophisticated teacher models to low-capacity student models, which has been widely deployed in various robotic applications, such as 3D object detection <cit.>, scene segmentation <cit.>, etc. Chen et al. <cit.> proposed a cross-modal knowledge distillation approach that transferred the spatial information from LiDAR to multi-camera BEV for 3D object detection. Hong et al. <cit.> applied knowledge distillation in monocular 3D object detection and extended it as a semi-supervised framework. Cen et al. <cit.> designed a bidirectional fusion network to learn the enhanced 3D representation and leveraged a knowledge distillation framework for improving performance. Although a number of applications made in robotics, there are still relatively few works on robotic grasping. In this work, we introduce distillation learning to transparent object grasping, aiming to utilize the knowledge from advanced teacher networks to distill an efficient student network with high accuracy.
	

§ METHOD

	

 §.§ Overview

	

	As shown in Fig. <ref>, our proposed model contains a teacher network and a student network. In the teacher network, we devise an encoder-decoder network where the encoder leverages position correlation blocks (PCB) to establish the correspondence between RGB images and depth maps, while the decoder adopts NeWCRFs <cit.> to restore the depth. In the student network, we use the same encoder-decoder network where the encoder utilizes Consistent feature correlation modules (CFCM) to capture pairwise relationships based on reliable positional information while the decoder uses the upsampling layers to recover the depth. Finally, a distillation loss with distance loss, structures loss, and edge loss is proposed to transfer the knowledge from the teacher network to the student network.
	
	

 §.§ Teacher Network

	The goal of our work is to improve the performance of synthetic data while keeping robustness to the real-world data. Thus, for the teacher network, we choose the Transformer network to extract spatial features since it performs better in capturing the overall structure of the objects. Specifically, we first adopt DeeplabV3+ to recognize transparent objects on RGB images and remove the corresponding areas on depth maps, which can avoid capturing wrong pairwise relationships between RGB images and depth maps. After that, patch embedding is employed on both channels respectively to split the images into patches and extract features. Because these features come from different channels and cannot provide useful mapping relationships for the network, we proposed our position correlation block to set up the correspondence between them.
	
	Position correlation block (PCB). Given a set of RGB image feature I and depth feature D, we first initialize the features by utilizing a layer normalization:
	
		
		
		
    x_i  = LN(I) , x_d = LN(D)

	where LN refers to layer normalization. In order to correlate the two different features, we introduce the attention layer. Since there are incomplete points and missing points in the depth map, we choose RGB images as query Q to search for the corresponding key K in RGB images and value V in the depth map. The attention layer can be denoted as follows:
	
		
		
		
    Q  = x_i , K = x_i , V=x_d  
    
    			F_qkv = SoftMax(Q· K^T+ B)· V  + I  
    
    			F_corr = MLP(LN(F_qkv)) + F_qkv

	where B is the relative position bias, MLP denotes a two-layer perceptron, F_qkv and F_corr represent the attention score and the correlated features. Although F_corr establishes the correspondence between RGB images and depth maps, it only considers the correlation of the regional patches and ignores the context information. Inspired by the Swin Transformer <cit.>, we adopt shifted window based attention to set up the connection between patches. Particularly, we apply the shifted window partitioning to the correlated features:
	
		
		
		
    I_shift = Shift(F_corr),D_shift = Shift(x_d)

	where Shift denotes shifted window partitioning, I_shift and D_shift represent the shifted RGB image features and the shifted depth features. Similar to the previous attention layer, the shifted image features are used to search for the corresponding key K in shift RGB image features and value V in the shifted depth features. Finally, a patch merging layer is adopted to reduce the height and width by half and expand the number of channels to twice. By stacking up four sets of PCBs, we can obtain the encoding features.
	
	In terms of the decoding stage, we follow <cit.> which develops a bottom-up structure that consists of four neural windows fully-connected Conditional Random Fields modules (NeWCRFs), where each module receives the features from the previous layer as well as the low-level features from the encoder through skip connections and outputs the upscale features utilizing a shuffle operation. Finally, the decoder outputs the completed depth map.
	

 §.§ Student Network

	The student network aims to keep pace with the teacher network while staying efficient at the same time. Therefore, for the student network, we choose CNN as our feature extractor because of its efficiency. Specifically, we first remove the depth of transparent objects by leveraging DeeplabV3+. Different from the teacher network that leverages RGB images to query corresponding positions on depth maps, our student network utilizes a more direct and efficient module called consistent feature correlation module which establishes correspondence based on the reliable positions of the RGB images and depth maps. The detail of the module is introduced as follows.
	
	Consistent feature correlation module (CFCM). Given a set of RGB image feature I and depth feature D, we set up two branches to calculate the consistency respectively, where each branch has the same structure except for the input. For clarity, we take the RGB image branch as an example. In particular, we first leverage a 1  × 1 convolution network to aggregate the different channel information and increase the number of channels. To avoid using too many convolutional layers that destroy the original spatial structure, the aggregated features are concatenated with the original depth features and sent into a 1  × 1 convolution network to compute the consistency score C_I. The consistency score C_I is formulated as follows:
	
		
		
		
    C_I = Conv_1 × 1( Conv_1×1(I) ⊕ D )

	where Conv_1×1 represents the  1  × 1 convolution network,⊕ indicates the channel concatenation, By multiplying the original RGB image feature I and the consistency score C_I, we can obtain the reliable RGB image and P_I.
	
		
		
		
    P_I = C_I⊙ I

	where ⊙ denotes element-wise multiplication. Following the same process, we can obtain a reliable depth map P_D. These two reliable visual maps are combined through element-wise addition to generate the integrated features F.
	
		
		
		
    F = P_I + P_D

	Finally, we devise an efficient CNN block to capture the pairwise relationships, where each block consists of a ReLU activation, a batch normalization, and a convolution layer to capture pairwise relationships. After stacking up four sets of CFCM, we can obtain the decoded features.
	
	In terms of the decoder, four upsampling blocks are deployed, where each block integrates a CNN network and an upsampling layer. Similarly, the encoded features are transferred to the decoder via skip connections, ensuring the multiscale features can be fully utilized. Finally, the decoder outputs the completed depth map.
	
	
	

 §.§ Distillation Loss

	Considering that the CNN-based student network can not acquire sufficient context features and leads to an unsatisfactory performance on the overall completion, we design our distillation loss into three components: distance loss, structural loss, and edge loss, where the distance loss is used to shrink the distance error, and the structural loss and edge loss ensure the overall structure of the completed depth.
	
	Distance Loss. Scale-Invariant Logarithmic (SILog) loss <cit.> has been proven to have the ability to capture scale-invariant depth information, thus we introduce it as our distance loss. Particularly, we first calculated the logarithm difference between the predicted depth map Δ E_p and the real depth Δ E_gt:
	
		
		
		
    Δ E_p =logD_s - logD_t , Δ E_gt =logD_s - logD_gt

	where D_s and D_t denote the predicted depth map completed by the student network and teacher network respectively. D_gt is the groundtruth depth value. For K valid values in the depth map, the SILog loss ℒ_d is computed as follows:
	
		
		
		
    ℒ_d    = α√(1/K∑_^Δ E_gt^2 -  λ/K^2 (∑_^Δ E_gt)^2)
        + β√(1/K∑_^Δ E_p^2 -  λ/K^2 (∑_^Δ E_p)^2)

	where α and β are scale constants and λ is the variance minimizing factor. In our experiments, α and β are set to 3 and 7 respectively, λ is set to 0.85.
	
	
	
	Structural Loss. We utilize the structural metrics from the structural similarity index measure (SSIM) and calculate their mean square error (MSE) as our structural loss. Specifically, the structural metric is defined as follows:
	
		
		
		
    C(x,y) = σ_x,y+θ/σ_xσ_y+ θ

	where x and y represent the depth maps, σ_x is the variance of the depth maps, σ_x,y is the covariance of the depth maps, and θ is the constant to prevent the dominator from being zero. In our work, we compute two sets of structural metrics, where the first metric C(D_gt,D_s) is computed between the groundtruth depth maps and the predicted depth maps from the student network, and the other metric C(D_t,D_s) is calculated between the predicted depth maps from the teacher network and the student network. Finally, the structural loss ℒ_s is computed as follows:
	
		
		
		
    ℒ_s = MSE(C(D_gt,D_t),C(D_t,D_s))

	where MSE denotes the mean square error.
	
	Edge Loss. Since the boundary of the transparent objects is always confused with the background, our edge loss is designed to capture the horizontal and vertical variations of the consecutive points. Particularly, the horizontal variations are defined as follows:
	
		
		
		
    V_x^r = |D_x,y^r - D_x+1,y^r| ⊙ M_x
where V_x^r denotes the horizontal variations of the depth maps r, D_x,y^r represents the depth maps r, M_x is the mask of the transparent objects in the horizontal direction. Similarly, the vertical variations are defined as follows:
	
		
		
		
    V_y^r = |D_x,y^r - D_x,y+1^r| ⊙ M_y

	where V_y^r denotes the vertical variations of the depth maps r and M_y is the mask of the transparent objects in the vertical direction. In our work, we compute two edge losses, where the first loss ℒ_e^gt,s is computed between the groundtruth depth maps and the predicted depth maps from the student network.
	
		
		
		
    ℒ_e^gt,s = MSE(V_x^gt,V_x^s) + MSE(V_y^gt,V_y^s)

	and the other ℒ_e^t,s is calculated between the predicted depth maps from the teacher network and the student network.
	
		
		
		
    ℒ_e^t,s = MSE(V_x^t,V_x^s) + MSE(V_y^t,V_y^s)

	
	In summary, the total loss of DistillGrasp ℒ is defined as follows:
	
		
		
		
    ℒ = ℒ_d + λ_1 *  ℒ_s + λ_2 *  ℒ_e^gt,s + λ_3 *   ℒ_e^t,s

	where λ_1, λ_2 and λ_3 are empirically set as 0.1, 0.3, and 0.7, respectively.
	
	

§ EXPERIMENTS

	

 §.§ Dataset

	We conducted our experiments on a large-scale transparent object dataset called ClearGrasp <cit.>, which contains 9 types of synthetic objects and 10 types of real-world objects with a total number of over 50k images. The objects are shown in Fig. <ref>. In this work, we follow the data divisions in Sajjan et al. <cit.> by using 5 known synthetic objects for training, and 5 overlapping real-world objects for testing. To verify the generalizability, 4 novel synthetic objects, and 5 novel real-world objects are used for testing.
	
	
	[table]
		labelsep=newline,
		singlelinecheck=true,
	
	[table]labelformat=simple, labelsep=newline, textfont=sc,justification=centering
	font=scriptsize
	
	
	

 §.§ Evaluation Metrics

	Following previous works  <cit.>, we adopt four common evaluation metrics including the root mean square error (RMSE), absolute relative difference (REL), mean absolute error (MAE), and accuracy with threshold δ_t. Particularly, the threshold t is set to 1.05, 1.10, and 1.25, where 1.05 is the strictest requirement for evaluating depth completion. Note that all these metrics are calculated based on the transparent object areas.
	
	

 §.§ Baselines

	We compare our teacher network (denoted as Our(T)) and student network (denoted as Our(S)) with recent deep learning methods as follows:
	
	ClearGrasp (CG) <cit.>, extracted multiple visual features from RGB-D images to infer the depths of transparent objects.
	
	DepthGrasp (DG) <cit.>, designed a self-attentive adversarial network with spectral residual blocks to capture the geometric information and generate the missing points.
	
	TransCG (DFNet) <cit.>, proposed an end-to-end depth completion network that uses U-Net architecture to extract fused features and outputs a refined depth map.
	
	Local Implicit Function (LIDF) <cit.>, leveraged a local implicit neural representation based on ray-voxel pairs to interface the depth maps.
	
	Fast Depth Completion (FDCT) <cit.>, introduced a fusion branch and cross-layer shortcuts to capture local information and designed a loss function to optimize the training process.
	

 §.§ Performance of the Approaches on Depth Completion

	Table <ref> summarizes the performance of different approaches on depth completion, from which we have some observations. 1) CG and DG achieve stable results in both synthetic and real-world environments, manifesting that the additional geometrics features can provide extra structural information in reducing the gap between synthetic data and real-world data. However, the direct concatenation of different visual features limits the performance of the methods. 2) DFNet and FDCT achieve excellent results on the synthetic known objects, but they perform poorly on real-world objects, potentially due to the architecture of CNNs that only consider the local information and are more likely to get overfitting. 3) LIDF performs excellent results in both synthetic and real-world environments, however, the offset ray-voxel pairs will affect the surrounding depth values, leading to regional bias. 4) Our teacher network achieves a competitive result on synthetic known objects and outperforms other methods in synthetic novel objects, real-known objects and real-novel objects, demonstrating the effectiveness and generalizability of our method. In particular, our network has achieved a maximum of 6% improvement on real known objects, which is beneficial to the architecture of the transformer network for capturing geometric structure and position correlation block for correlating positional information. 5) Although there is a gap between the student network and the teacher network, the student network still achieves promising results on the novel objects. We believe it is because the teacher network effectively distills the structure and the edge information from the RGB images and depth maps and successfully teaches them to the student network.
	
	In Fig. <ref>, we visualize some examples of completed depth maps and highlight the generated details with red circles. From the figures we can observe that both our teacher network and student network can complete the depth map with clear shape and contour, but the student network still needs to improve the completion of object boundaries.
	
	

 §.§ Speeds of the Approaches on Depth Completion

	

	To simulate the real-world deployment, we use real novel objects for testing the time cost. Table <ref> lists the time cost and parameters size of different approaches, from which we can observe that FDCT uses the least time to complete the depth map, however, its performance is not promising. Although our teacher network can achieve the best results in completing the depth map, it sacrifices twice as much time. Unlike the previous two networks, our student network achieves an ideal balance in time and accuracy, where it only sacrifices a little accuracy to acquire higher efficiency, reaching the speed of 48 FPS.

	

	

 §.§ Effect of Different Sets of Layers

	As one of the important hyperparameters, the number of layers plays an important role in the results. Thus, in Table <ref> we summarize the performance by choosing different sets of layers. Note that we use Teacher_X and Student_X to denote different types of networks, where X denotes the number of layers. From the table, we can notice that when the number of layers is set to 4, the network can achieve the best results. While using fewer or more layers, the performance will drop a little bit. We believe that it is because when using fewer sets of encoders, the network cannot further extract the details from the RGB images and depth maps, leading to lower accuracy. While using more encoders, the features will be compressed very small, where the spatial structure of the object will be violated. Moreover, the small size of features will increase the difficulties for the network to recover the depth maps.

	
	
	

 §.§ Ablation Study

	We design a series of ablation experiments to verify each component. For the teacher network, we adopt self-attention mechanisms based on RGB images and depth maps respectively to replace the position correlation block (PCB) and denote them as TCH_SA(RGB) and TCH_SA(Depth), respectively. For the student network, we denote the student network without consistent feature correlation module (CFCM) as STU_CFCM, the student network without distillation loss as STU_DL, the student network without CFCM and distillation loss as STU_CFCM_DL, and student network without teacher network as STU_Alone. Note that we use common loss RMSE to replace the distillation loss and adopt UNet as our baseline model. Additionally, we further verify the effect of each distillation loss.
	
	Table <ref> shows the experimental results, from which we can find that our PCB in the teacher network achieves a maximum 18.2% improvement compared with the single input of self-attention, illustrating that the correlation of features can facilitate depth completion. In the student network, the employment of CFCM and distillation loss obtain a certain improvement in accuracy, where the CFCM has a greater impact on the depth completion because it fuses the two different types of features and decides the entire quality of the depth completion, while distillation loss only plays a part in optimizing the generation of edge and object structure. 
	
	Intuitively, we visualize some examples of the completed depth maps generated by the ablation experiments and highlight the generated details with red circles. The results are shown in Fig. <ref>. From Fig. <ref>(a) we can observe that our position correlation block can better deal with the complex shapes objects owing to the correspondence established between RGB images and depth maps. From Fig. <ref>(b) we can notice that the adoption of CFCM has a better result on the overall depth completion and the usage of distillation loss can improve the generation of the object structure and contour.
	
	
	Table <ref> illustrates the effect of each distillation loss, from which we can notice that the distance loss plays a critical role in the performance as it computes the point-wise errors and decides the overall completion. The structure loss and edge loss determine the object structure and boundary details respectively, which improve the results to a certain extent. The combination of loss functions achieves the best results since it guarantees depth completion in overall effect and details.

	

	

 §.§ Robot Grasping

	

	In order to demonstrate the applicability of our network in practical tasks, we deployed our system on a single-arm robot UR10e to grasp real-world transparent objects. Specifically, we choose the GR-CNN <cit.>, which was verified by the previous work <cit.>, as our grasping method. Fig. <ref>(a) presents 8 novel transparent objects that do not appear in the ClearGrasp dataset for grasping, including  “Corrugated collins glass”, “Goblet”, “Rocks glass”, “Corrugated round glass”, “Hexagon cup”, “Handler cup”, “Cylindrical cup” and “Seasoning pot”. For each object, the robot will attempt to grasp it 10 times, and only when the object is held for more than 10 seconds will be considered successful. 
	
	Table <ref> lists the success rate of grasping, from which we can observe that our proposed network can tremendously improve the success rate compared with using raw depth maps and outperform the state-of-the-art method DepthGrasp in real-world grasping. Fig. <ref>(b) shows a few snapshots of UR10e grasping transparent objects, from which we can see that the robot can successfully hold the transparent objects, demonstrating the effectiveness of our proposed method.
	
	
	
	

§ CONCLUSION

	In this paper, we propose a novel distillation network called DistillGrasp for completing the depth of transparent objects. Specifically, in the teacher network, we proposed a position correlation block to search for the positional correspondence between RGB images and depth maps. For the student network, we designed an efficient consistent feature correlation module to capture the pairwise relationship based on reliable positional information. In order to transfer the overall knowledge from the teacher branch to the student branch, we present a distillation loss that takes distance loss, object structure, and edge information into consideration. Extensive experiments demonstrated that our teacher network achieves state-of-the-art performance on accuracy and generalization while the student network achieves ideally balanced results in terms of efficiency and speed. Moreover, the deployment of our system on a robot effectively verifies the applicability of our system.
	
	
	

§ ACKNOWLEDGMENT

	This work is supported by the National Natural Science Foundation of China (No. 91748107), the Special Research Fund (BOF) of Hasselt University (No. BOF23DOCBL11), the foundation of State Key Laboratory of Public Big Data(No. PBD2023-11), the Guangdong Innovative Research Team Program (No. 2014ZT05G157). Chen Junhong was sponsored by the China Scholarship Council (No. 202208440309).
	
	
	ieeetr
	40
		ref1
		J. Weibel, P. Sebeto, S. Thalhammer, and M. Vincze, “Challenges of Depth Estimation for Transparent Objects,” International Symposium on Visual Computing, pp. 277-288, 2023.
		ref4
		S. Sajjan, M. Matthew, M. Pan, N. Ganesh, J. Lee, A. Zeng, and S. Song, “ClearGrasp:3D shape estimation of transparent objects for manipulation,” IEEE International Conference on Robotics and Automation (ICRA), pp. 1-13, 2020.
		ref5
		Y. Tang, J. Chen, Z. Yang, Z. Lin, Q. Li, and W. Liu, “Depthgrasp: Depth completion of transparent objects using selfattentive adversarial network with spectral residual for grasping,” IEEE International Conference on Intelligent Robots and Systems (IROS), pp. 5710-5716,2021.
		ref6
		K. Chen, S. Wang, B. Xia, D. Li, Z. Kan, and B. Li, “TODETrans: Transparent Object Depth Estimation with Transformer,” IEEE International Conference on Robotics and Automation (ICRA), pp. 4880-4886, 2023.
		ref7
		Y. Hong, J. Chen, Y. Cheng, Y. Han, F. Van Reeth, L. Claesen, and W. Liu, “Cluedepth grasp: Leveraging positional clues of depth for completing depth of transparent objects,” Frontiers in Neurorobotics, vol. 16, 2022.
		ref8
		J. Jiang, G. Cao, J. Deng, T. -T. Do and S. Luo, "Robotic Perception of Transparent Objects: A Review," in IEEE Transactions on Artificial Intelligence, vol. 5, no. 6, pp. 2547-2567, 2023.
		ref9
		U. Klank, D. Carton, and M. Beetz, “Transparent object detection and reconstruction on a mobile platform,” IEEE International Conference on Robotics and Automation (ICRA), 2011, pp. 5971–5978.
		ref11
		J. Ichnowski, Y. Avigal, J. Kerr, and K. Goldberg, “Dex-nerf: Using a neural radiance field to grasp transparent objects,” Conference on Robot Learning (CoRL), 2021.
		ref12
		J. Kerr, L. Fu, H. Huang, J. Ichnowski, M. Tancik, Y. Avigal, A. Kanazawa, and K. Goldberg, “Evo-nerf: Evolving nerf for sequential robot grasping,” Conference on Robot Learning (CoRL), 2022.
		ref13
		Y. Qian, M. Gong, and Y. H. Yang, “3d reconstruction of transparent objects with position-normal consistency,” in Proc. IEEE/CVFInternational Conference on Pattern Recognition(ICPR), pp. 4369–4377, 2016.
		ref19
		Hinton, Geoffrey, Oriol Vinyals, and Jeff Dean, “Distilling the knowledge in a neural network,” arXiv preprint arXiv:1503.02531, 2015.
		ref32
		Z. Chen, Z. Li, S. Zhang, L. Fang, Q. Jiang, and F. Zhao, "Bevdistill: Cross-modal bev distillation for multi-view 3d object detection," International Conference on Learning Representations (ICLR), 2023.
		ref33
		Y. Hong, H. Dai, and Y. Ding, "Cross-modality knowledge distillation network for monocular 3d object detection," in European Conference on Computer Vision. Springer, 2022, pp. 87–104.
		ref34
		J. Cen, S. Zhang, Y. Pei, K. Li, H. Zheng, M. Luo, Y. Zhang, and Q. Chen, "Cmdfusion: Bidirectional fusion network with cross-modality knowledge distillation for lidar semantic segmentation," IEEE Robotics and Automation Letters, vol. 9, no. 1, pp. 771–778, 2023.
		ref35
		F. Jiang, H. Gao, S. Qiu, H. Zhang, R. Wan, and J. Pu., "Knowledge Distillation from 3D to Bird’s-Eye-View for LiDAR Semantic Segmentation," in 2023 IEEE International Conference on Multimedia and Expo (ICME), Brisbane, Australia, 2023 pp. 402-407.
		ref24
		W. Yuan, X. Gu, Z. Dai, S. Zhu, and P. Tan, “Neural Window Fully-connected CRFs for Monocular Depth Estimation,”  IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3906-3915, 2022.
		ref26
		Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” IEEE/CVF International Conference on Computer Vision (ICCV),  pp. 10012-10022, 2021.
		ref27
		D. Eigen, C. Puhrsch, and R. Fergus, “Depth map prediction from a single image using a multi-scale deep network,” Neural Information Processing Systems(NIPS), pp. 2366-2374, 2014.
		ref28
		H. Fang, H. -S. Fang, S. Xu and C. Lu, “TransCG: A Large-Scale Real-World Dataset for Transparent Object Depth Completion and a Grasping Baseline,” IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 7383-7390, 2022.	
		ref31
		L. Zhu, A. Mousavian, Y. Xiang, H. Mazhar, J. Eenbergen, S. Debnath, and D. Fox, “RGB-D Local Implicit Function for Depth Completion of Transparent Objects,” IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),  pp. 4647-4656, 2021.
		ref29
		T. Li, Z. Chen, H. Liu, and C. Wang, “FDCT: Fast Depth Completion for Transparent Objects,”  IEEE Robotics and Automation Letters, vol. 8, no. 9, pp. 5823-5830, 2023.
		ref30
		S. Kumra, S. Joshi, and F. Sahin, “Antipodal robotic grasping using generative residual convolutional neural network,” IEEE International Conference on Intelligent Robots and Systems (IROS), pp. 9626-9633, 2020.
	
	


