# Can We Treat Noisy Labels as Accurate?

Yuxiang Zheng1 Zhongyi Han211footnotemark: 1 22footnotemark: 2 Yilong Yin3 Xin
Gao2 Tongliang Liu1
1Sydney AI Center, The University of Sydney;
2CEMSE, King Abdullah University of Science and Technology;
3School of Software, Shandong University  Equal contributions.Corresponding
author.

###### Abstract

Noisy labels significantly hinder the accuracy and generalization of machine
learning models, particularly due to ambiguous instance features. Traditional
techniques that attempt to correct noisy labels directly, such as those using
transition matrices, often fail to address the inherent complexities of the
problem sufficiently. In this paper, we introduce EchoAlign, a transformative
paradigm shift in learning from noisy labels. Instead of focusing on label
correction, EchoAlign treats noisy labels ($\tilde{Y}$) as accurate and
modifies corresponding instance features ($X$) to achieve better alignment
with $\tilde{Y}$. EchoAlign’s core components are (1) EchoMod: Employing
controllable generative models, EchoMod precisely modifies instances while
maintaining their intrinsic characteristics and ensuring alignment with the
noisy labels. (2) EchoSelect: Instance modification inevitably introduces
distribution shifts between training and test sets. EchoSelect maintains a
significant portion of clean original instances to mitigate these shifts. It
leverages the distinct feature similarity distributions between original and
modified instances as a robust tool for accurate sample selection. This
integrated approach yields remarkable results. In environments with 30%
instance-dependent noise, even at 99% selection accuracy, EchoSelect retains
nearly twice the number of samples compared to the previous best method.
Notably, on three datasets, EchoAlign surpasses previous state-of-the-art
techniques with a substantial improvement.

##  1 Introduction

The rapid advancement of neural networks highlights the importance of learning
from noisy labels (LNL) (Tan and Le, 2019; Dosovitskiy et al., 2021; Stiennon
et al., 2020; Chen et al., 2023a). Though cost-effective for large datasets,
web crawling, and crowdsourcing can introduce noisy labels that hinder machine
learning model generalization (Yu et al., 2018; Li et al., 2017; Welinder et
al., 2010; Zhang et al., 2017; Natarajan et al., 2013; Gu et al., 2023). For
example, recent studies indicate this label noise in pretraining data impacts
out-of-distribution generalization of foundation models on downstream tasks
(Chen et al., 2023a, 2024). Noisy labels are categorized as random, class-
dependent, or instance-dependent, with the latter two being particularly
challenging due to ambiguous instance features. This ambiguity complicates the
distinction between mislabeled examples and those belonging to other classes
(Menon et al., 2018; Xia et al., 2020; Yao et al., 2023a; Bai et al., 2023).

Prior work has largely approached LNL from either noise-modeling-free or
noise-modeling standpoints. The noise-modeling-free methods, such as
extracting examples with small losses (Han et al., 2018; Yu et al., 2019; Wang
et al., 2019), can only do clean sample selection and cannot correct noisy
labels to clean labels, which waste much supervision information. On the other
hand, noise-modeling approaches explicitly consider the label-noise generation
process (Scott et al., 2013; Scott, 2015; Goldberger and Ben-Reuven, 2016).
These methods operate on the principle that the clean class posterior can be
deduced using the noisy class posterior, derived from noisy data, in
conjunction with a transition matrix (Berthon et al., 2021). The concept is
that, given a transition matrix, an optimal classifier mirroring clean data’s
behavior can be trained solely using sufficient noisy data (Reed et al., 2014;
Liu and Tao, 2015). Nonetheless, accurately modeling the general process,
specifically the transition matrix, is ill-posed by only exploiting noisy data
(Xia et al., 2019; Cheng et al., 2020). The additional assumptions required
for this model are often hard to validate and may not hold in real-world
datasets, as indicated by recent studies (Xia et al., 2020; Yao et al., 2023b;
Liu et al., 2023). This mismatch frequently results in an inability to
eliminate label noise thoroughly.

Figure 1: Instance modification effectively aligns instances with their
labels, while label correction struggles with ambiguous cases.

In this paper, we challenge the conventional approach to label noise with
_instance modification_. Instead of attempting to correct noisy labels, we
strategically adjust instances to better align with their labels, even if
those labels are incorrect. This groundbreaking new direction of LNL is
visually depicted in Figure 1. At its core, instance modification tackles the
underlying cause of label noise. Drawing on principles from causal learning
(Neuberg, 2003; Peters et al., 2017; Yao et al., 2021), we view the generation
of instance-dependent label noise through a causal lens, as illustrated in the
causal graph of Figure 2. Here, the observed noisy labels ($\tilde{Y}$) emerge
from the instances ($X$), influenced by latent variables ($Z$). In
crowdsourcing scenarios, the difficulty of labeling blurred or ambiguous
instances often leads to errors. Instead of trying to guess the ‘true’ label
in these cases, modifying the instance itself to make it more easily
classifiable can be more effective. Therefore, instance modification offers a
practical solution for particularly challenging instances that are difficult
to label correctly – those that are inherently ambiguous for both humans and
machines.

While instance modification offers a breakthrough, it presents significant
challenges at both the instance and dataset levels (More discussions can be
found in the Appendix E). At the instance level, a critical challenge lies in
maintaining the instance’s core characteristics. Modifying instances,
particularly those with incorrect labels, to strictly align with potentially
unrelated labels can lead to substantial characteristic shifts. Imagine
modifying a dog image to resemble an airplane – the resulting image might be
unnatural or distorted, compromising the dog’s intrinsic characteristics like
posture, contour, and depth information. Meanwhile, it’s also crucial to
preserve these intrinsic characteristics during modification for instances
with correct labels. At the dataset level, instance modification can induce
covariate shifts between training and test sets (Han et al., 2022a, b). These
covariate shifts can hinder a model’s ability to generalize effectively to
unseen real-world data.

To tackle these challenges, we propose a simple yet effective framework
EchoAlign (§4). EchoAlign has two key components: EchoMod and EchoSelect.
EchoMod modifies instances using controllable generative models to align them
with noisy labels while preserving their intrinsic characteristics. By
controlling the image generation process using the original image and noisy
label as guiding information, EchoMod ensures the overall quality and
consistency of the training data. EchoSelect addresses the issue of covariate
shift by strategically selecting original instances with correct labels to
balance the distribution between original and modified instances in the
training data. This selection leverages a novel insight: after instance
modification, the Cosine feature similarity distribution between original and
modified images shows a clear distinction between clean and noisy samples.
EchoSelect employs this insight through a unique similarity metric to select
the most reliable training instances to reconstruct the learnable set for
further supervised or self-supervised training.

Our contributions and the key findings are summarized as follows: (1) We
introduce a transformative change in how noisy labels are approached. Instead
of focusing on correcting them, we advocate treating noisy labels as accurate
and modifying instances to align with them. This paradigm shift is supported
by our theoretical analysis (§3). (2) We present EchoAlign, a framework
specifically designed to address the challenges of instance modification. It
employs EchoMod for controlled instance modification and EchoSelect for
strategic sample selection (§4). (3) We empirically validate the advantages of
instance modification and demonstrate EchoAlign’s exceptional performance in
noisy environments. Across three datasets with diverse noise types, EchoAlign
consistently outperforms state-of-the-art techniques, achieving significant
accuracy gains (§5).

##  2 Related Work

Learning with Noisy Labels Research in this domain has predominantly followed
two paths: (1) Noise-modeling-free methods: These methods primarily rely on
the memorization effects observed in deep neural networks (DNNs). They
capitalize on the tendency of DNNs to prioritize learning simpler (clean)
examples before memorizing more complex (noisy) ones (Arpit et al., 2017; Wu
et al., 2020; Kim et al., 2021). Techniques include early stopping (Han et
al., 2018; Nguyen et al., 2020; Liu et al., 2020; Xia et al., 2021; Lu et al.,
2022; Bai et al., 2021), pseudo-labeling (Tanaka et al., 2018), and leveraging
Gaussian Mixture Models in a semi-supervised learning context (Li et al.,
2020). (2) Noise-modeling methods: This category of approaches focuses on
estimating a noise transition matrix, which models how clean labels might be
corrupted into noisy observations. However, accurately modeling the noise
process is challenging when relying solely on noisy data (Xia et al., 2019;
Cheng et al., 2020). Existing studies rely on assumptions that might not hold
in real-world datasets (Xia et al., 2020; Yao et al., 2023b; Liu et al.,
2023). Consequently, these methods often struggle to fully eliminate label
noise, especially when dealing with structured dependent noise patterns, such
as subclass-dominant label noise (Bai et al., 2023).

Generative Models Recent advances in generative models, including variational
auto-encoders, generative adversarial networks, and diffusion models, have
transformed applications with their exceptional sample generation capabilities
(Du et al., 2023; Wang et al., 2023; Franceschi et al., 2023). Diffusion
models, particularly known for their superior output control, delicately
denoise signals (Zhang et al., 2023; Kingma et al., 2021). While these models
hold promise for noisy label scenarios, existing approaches like Dynamics-
Enhanced Generative Models (DyGen) (Zhuang et al., 2023) and Label-Retrieval-
Augmented Diffusion Models (Chen et al., 2023b) still focus on enhancing
predictions or retrieving latent clean labels. Our work introduces a
fundamentally different approach. We adopt controllable generative models,
treating noisy labels as correct and aligning instances to them. This bypasses
the challenges of traditional noise modeling and focuses on improving the
training data itself. Controllable generative models are a powerful subset of
generative models that allow for precise control over the generated outputs,
such as ControlNet (Zhang et al., 2023), iPromptDiff (Chen et al., 2023c).
Unlike traditional generative models, which generate images based on random
noise, controllable generative models take control information (_e.g._ , text
descriptions, class labels, or reference images) as input (Bose et al., 2022).
This control information guides the generation process, ensuring that the
generated outputs align with the desired characteristics.

##  3 Analysis

Problem Definition In addressing the challenges posed by learning from noisy
labels (LNL), we formally define the problem and introduce the concept of
instance modification within a mathematical framework. Let $\mathcal{X}$
represent the input space of instances and $\mathcal{Y}$ the space of labels.
In the traditional LNL setting, each instance $X\in\mathcal{X}$ is associated
with a noisy label $\tilde{Y}\in\mathcal{Y}$, which may differ from the true
label $Y\in\mathcal{Y}$. The goal is to learn a mapping
$f:\mathcal{X}\rightarrow\mathcal{Y}$ that predicts the true label $Y$ as
accurately as possible, despite the presence of noisy labels. Instance
modification diverges from the conventional approach of directly correcting
noisy labels $\tilde{Y}$ to match the true labels $Y$. Instead, it proposes
adjusting each instance $X$ to better align with its given noisy label
$\tilde{Y}$.

Figure 2: A graphical causal model, revealing a data generative process with
instance-dependent label noise.

Mathematically, this involves transforming each instance $X$ into a modified
instance $X^{\prime}$, such that $f(X^{\prime})$ more closely approximates
$\tilde{Y}$, leveraging the inherent information contained within the noisy
label itself.

Theoretical Analysis According to the causal learning framework (Liu et al.,
2023; Yao et al., 2021), the noise can often be represented as a function of
both the instance features and external factors, encapsulated by latent
variables $Z$. We assume the causal relations (commonly occurring in
crowdsourcing scenarios) are represented in the causal graph as illustrated in
Figure 2, where $Z$ represents latent variables that affect both $X$ and
$\tilde{Y}$ indirectly through $X$. Instance modification aims to transform
$X$ into $X^{\prime}$ such that the modified instance $X^{\prime}$ aligns
better with $\tilde{Y}$ under the assumption that $\tilde{Y}$ contains some
information about the true label $Y$. Accordingly, we can deduce the
effectiveness of instance modification as follows.

######  Theorem 3.1 (Effectiveness of Instance Modification).

Assume that the noisy labels are generated by a stochastic process influenced
by latent variables $Z$, where $\tilde{Y}=h(Y,Z)$ and $Y$ are the true labels.
Let $T$ be a transformation such that $X^{\prime}=T(X,\tilde{Y};\theta)$,
where $\theta$ is chosen to optimize the alignment of $X^{\prime}$ with
$\tilde{Y}$. Then, under this transformation, the predictive performance of a
model trained on $(X^{\prime},\tilde{Y})$ is theoretically improved compared
to a model trained on $(X,\tilde{Y})$ in terms of:

  1. 1.

Alignment: The mutual information between $X^{\prime}$ and $\tilde{Y}$,
$I(X^{\prime};\tilde{Y})$, is maximized relative to $I(X;\tilde{Y})$,
indicating better alignment of modified instances with their noisy labels.

  2. 2.

Error Reduction: Compared to a model trained on the original instances $X$,
the expected prediction error
$\mathbb{E}_{X^{\prime},Y}[(Y-f(X^{\prime}))^{2}]$ is minimized, where $f$ is
the prediction function trained using the modified instances $X^{\prime}$.
This assumes that the distribution of $X^{\prime}$ does not deviate
significantly from the distribution of $X$, ensuring that the learned model
generalizes well to the original distribution.

  3. 3.

Estimation Stability: The variance of the estimator $f$ is reduced when using
$X^{\prime}$ compared to $X$, resulting in more stable predictions.

  4. 4.

Generalization: Modifications in $X^{\prime}$ lead to better generalization.
By transforming the original instances to better align with their noisy
labels, the model trained on $X^{\prime}$ is less to overfit to the noise and
more capable of capturing the true underlying patterns in the data.

This improvement is contingent upon the assumption that the noise model $h$
and the transformation $T$ are appropriately defined and that the latent
variable model adequately captures the underlying causal structure of the
data. More details and proofs can be found in the Appendix A.

This theoretical framework 3.1 suggests that instance modification, by
aligning more closely with noisy but informative labels $\tilde{Y}$, can
leverage the inherent structure and causality in the data to enhance learning.
It demonstrates that instance modification improves alignment between
instances and noisy labels, reduces information loss, and ultimately leads to
better generalization. These insights provide several key motivations for the
design of our method. Firstly, the improvements in alignment highlight the
importance of modifying instances to embed noisy label information directly.
This motivates the use of controllable generative models in EchoAlign, which
can effectively incorporate label information into the instance features.
Secondly, ensuring a minimal distribution difference between $X$ and
$X^{\prime}$ is crucial. EchoMod generates $X^{\prime}$ with small
distribution differences from $X$, while EchoSelect retains clean samples to
control distribution differences, ensuring better generalization on test data.

Figure 3: The feature similarity between the original and modified instances
is a valuable metric for sample selection after instance modification.

Thirdly, the improvement in estimation stability indicates that using modified
features can result in more consistent and reliable model predictions, driving
the focus on preserving the essential characteristics of the data during
transformation to reduce variability and enhance stability.

Analyzing Feature Similarity Distributions In this study, we address the
challenges of instance modification which can induce distribution shifts
between the training and test sets. Preserving clean original instances is
crucial to mitigating these shifts. Existing sample selection methods (_e.g._
, small loss (Han et al., 2018)) often falter under complex label noise
conditions, such as instance-dependent noise, necessitating a more precise
selection strategy. To this end, we find an interesting phenomenon: Clean
samples generally exhibit higher similarity between features of original and
modified images, indicating minimal semantic and label changes after
modification, whereas noisy samples display lower similarity due to
significant semantic and label adjustments. Utilizing the feature similarity
distributions between original and modified instances emerges as a robust tool
for enhancing sample selection accuracy. These distinctions are visually
represented in Figure 3. The similarity is computed using the CLIP ViT-B-32
feature extractor (Radford et al., 2021) on the CIFAR-10 dataset with 30%
instance-dependent noise. We use ControlNet (Zhang et al., 2023) to modify
instances. The black dashed line indicates the sample threshold achievable by
the previous best method at 96% accuracy (Yang et al., 2022). In contrast,
EchoSelect, at 96% accuracy, can retain the samples in the yellow section. In
environments with 30% instance-dependent noise, EchoSelect retains nearly
twice the number of samples at 99% accuracy. Statistical validation using the
Kolmogorov-Smirnov test confirmed significant differences in the distributions
(p-value < 0.001), demonstrating the utility of feature similarity as a robust
metric for identifying clean samples within noisy datasets.

##  4 EchoAlign

The EchoAlign framework tackles the challenge of noisy labels in supervised
learning. It comprises two primary components: (1) EchoMod modifies data
instances to align them better with their potentially incorrect labels. (2)
EchoSelect selects the most reliable modified instances, creating a cleaner,
more consistent training dataset, which is used to train a general neural
network.

###  4.1 EchoMod: Instance Modification

Motivation When labels are noisy, they do not reflect the true characteristics
of the corresponding data instances. This discrepancy can hinder a model’s
ability to learn meaningful patterns. EchoMod addresses this by transforming
data instances to be more consistent with their noisy labels. This controlled
modification helps the model extract relevant information when the labels are
noisy.

Mechanism EchoMod leverages a pre-trained controllable generative model
(_e.g._ , a controllable diffusion-based model) to modify data instances. The
primary goal is to enhance the alignment between instances and their
potentially noisy labels. This alignment is achieved by carefully guiding the
generative model’s process. Firstly, the controllable generative model has
undergone prior training on a large dataset. This pre-training has equipped
the model with a deep understanding of the patterns and structures inherent in
the data domain. Secondly, EchoMod provides both the original instance ($X$)
and the noisy label ($\tilde{Y}$) as inputs to the generative model. This dual
conditioning shapes the output, encouraging the model to produce a modified
instance ($X^{\prime}$) that closely aligns with the noisy label while still
retaining essential characteristics of the original data. Striking this
balance between label alignment and preventing excessive distortion is
crucial.

Reasonableness, Effectiveness, and Fairness Handling label noise in noisy
label learning is a well-recognized challenge. Previous works have primarily
focused on the utilization and optimization of internal data. Our approach
introduces a novel perspective by integrating external knowledge to enhance
model robustness. This integration does not compromise fairness, as our
flexible framework accommodates various generative models, and can be fine-
tuned for specific noisy label problems. By doing so, we ensure that the
method does not overly rely on any particular model. This approach is
particularly advantageous when dealing with ambiguous data. The inherent
ambiguity in the data can lead to low confidence in direct discrimination.
Instead, by modifying the input data through controllable generative models,
we can better resolve discrepancies between instances and labels while
preserving the meaningful characteristics of the original data. This not only
improves the model’s discriminative capability but also enhances overall
performance and reliability.

No Fine-Tuning for Generalizability In most cases, EchoMod can leverage a pre-
trained controllable generative model without fine-tuning during the alignment
process. This preserves the model’s ability to understand general data
characteristics, promoting EchoAlign’s applicability across various domains.
While EchoMod can be effective without fine-tuning, additional performance
gains might be realized by tailoring the controllable generative model to
highly specialized tasks or data distributions. In such cases, fine-tuning the
parameters ($\theta$) could lead to better alignment between instances and
noisy labels. Examples of this specialization include medical imaging or
scientific data.

###  4.2 EchoSelect: Instance Selection

Motivation While EchoMod improves the alignment between instances and noisy
labels, some modified instances may still exhibit inconsistencies.
Additionally, the instance modification process can introduce distribution
shifts between the modified training data and the true test distribution.
EchoSelect safeguards against these issues by identifying and retaining only
the most reliable modified instances. This filtering enhances model
robustness, reduces the impact of noisy data, and helps mitigate the
distribution shifts caused by instance modification.

Mechanism EchoSelect employs a metric to assess the similarity between
modified instances and a reference representation of clean data. We use the
Cosine similarity between feature vectors extracted using a suitable feature
extractor (_e.g._ , the image encoder of CLIP (Radford et al., 2021)):

$S(X^{\prime},X)=\frac{\bm{z}(X^{\prime})\cdot\bm{z}(X)}{\|\bm{z}(X^{\prime})\|\|\bm{z}(X)\|}\,,$
(1)

where $X^{\prime}$ and $X$ are modified and original instances, and $\bm{z}$
denotes the feature extractor.

Selection Process EchoSelect calculates similarity for all modified instances,
comparing them to their original counterparts. To mitigate distribution
shifts, priority is given to maintaining clean original instances as much as
possible. The final training set consists of two parts: (1) Original instances
with similarity larger than a determined threshold are deemed sufficiently
aligned with clean data characteristics and retained, and (2) modified
instances with similarity below the threshold are included. These instances
are likely those where the modification was most beneficial in aligning them
with the noisy labels, while also indicating some degree of difference from
the original distribution. This threshold $\tau$ balances the inclusion of
modified instances with the preservation of the original data distribution.
This threshold ensures that only instances aligned with the characteristics of
clean data are retained. Our sensitivity analysis (§5.3) verified the
robustness of $\tau$ across types of noise.

###  4.3 EchoAlign: Optimized Combination

Algorithm 1 EchoAlign Framework

0: Pre-trained controllable generative model $f_{\theta}$, Noisy dataset
$(X,\tilde{Y})$, Threshold $\tau$, Feature Extractor

0: Refined training dataset

1: Generate modified instances: $X^{\prime}\leftarrow f_{\theta}(X,\tilde{Y})$

2: Compute similarity: $S(X^{\prime},X)$ using Equation (1)

3: # Construct a refined dataset with two parts

4: Part 1: Original Instances

5: Select original instances where $S(X^{\prime},X)\geq\tau$

6: Part 2: Modified Instances

7: Select modified instances where $S(X^{\prime},X)<\tau$

8: Combine Part 1 and Part 2 to form the refined dataset

9: Return the refined dataset

The integration of EchoMod and EchoSelect enables the creation of a refined
training dataset that is aligned with noisy labels and filtered for quality.
This optimized dataset is better suited for robust learning in the presence of
large label noise. Since the refined training dataset can be further used to
train a supervised or self-supervised model for LNL, EchoAlign can be combined
with advanced LNL methods to address the impact of label noises further. The
integration of EchoMod and EchoSelect is encapsulated in the following
Algorithm 1, which details the steps for modifying instances and selecting the
optimal subset.

##  5 Experiments

###  5.1 Experiment Setup

Dataset Our experiments are conducted on two synthetic datasets: CIFAR-10, and
CIFAR-100 (Krizhevsky et al., 2009), and a real-world dataset: CIFAR-10N (Wei
et al., 2022). CIFAR-10 and CIFAR-100 each contain 50,000 training and 10,000
testing images, with a size of 32$\times$32, covering 10 and 100 classes
respectively. CIFAR-10N utilizes the same training images from CIFAR-10 but
with labels re-annotated by humans. Following previous research protocols (Bai
et al., 2021; Xia et al., 2019, 2023b), we corrupted these synthetic datasets
by three types of label noise. Specifically, symmetric noise randomly alters a
certain proportion of labels to different classes to simulate random labeling
errors; pair flip noise changes labels to specific adjacent classes with a
certain probability; instance-dependent noise modifies labels based on the
image features to related incorrect classes. Due to resource constraints, we
use a light and representative human-annotated noisy dataset CIFAR-10N instead
of Clothing1M (Xiao et al., 2015). A detailed runtime analysis is discussed in
the Appendix D, demonstrating that the runtime is reasonable for all datasets
used. For CIFAR-10N, we provide four noisy label sets: ‘Random i=1, 2, 3’,
each representing the label provided by one of three independent annotators;
and ‘Worst’, which selects the noisiest label when incorrect annotations are
present.

Baseline We compare EchoAlign against various paradigms of baselines for
addressing label noise. Under the robust loss function paradigm, we include
APL (Ma et al., 2020), PCE (Menon et al., 2019), AUL (Zhou et al., 2023), and
CELC (Wei et al., 2023) as baselines; under the loss correction paradigm, we
utilize T-Revision (Xia et al., 2019) and Identifiability (Liu et al., 2023);
under the label correction paradigm, we select Joint (Tanaka et al., 2018);
and under the sample selection paradigm, we employ Co-teaching (Han et al.,
2018), SIGUA (Han et al., 2020) and Co-Dis (Xia et al., 2023a). We compare
these methods against a simple cross-entropy (CE) loss baseline. Following the
fair baseline design used by Xia et al. (2023b), we do not compare with some
methods such as MixUp (Zhang et al., 2018), DivideMix (Li et al., 2020), and
M-correction (Arazo et al., 2019), as these involve semi-supervised learning,
making such comparisons unfair due to inconsistent settings.

Implementation Details All experiments were conducted on an NVIDIA V100 GPU
using PyTorch. The model architectures and parameter settings were kept
consistent with previous studies (Bai et al., 2021). The experiments were
configured with a learning rate of 0.1, using the Stochastic Gradient Descent
(SGD) optimizer with a momentum of 0.9, and weight decay set to $1\times
10^{-4}$. We applied 30% and 50% symmetric noise as well as 45% pair flip
noise on the CIFAR-10 and CIFAR-100 datasets to assess model performance. The
CIFAR-10 dataset utilized the standard ResNet-18 (He et al., 2016)
architecture, while CIFAR-100 used ResNet-34. Similarly, for the CIFAR-10N
dataset, the same ResNet-18 model was used. Prior to the training phase,
ControlNet was utilized as our reference model in the controllable generation
model module. This choice was strategic; ControlNet was the least effective
model identified in prior analyses (Chen et al., 2023c). Employing this model
underscores the robustness of our approach, ensuring that the efficacy of our
method is not overly contingent upon the capabilities of any specific
generative model. This decision highlights our method’s adaptability and
general efficacy across varying scenarios. We employed the Canny edge detector
as a simple preprocessor to extract features from the instances, using labels
as textual controls with the prompt "a photo of {label}". This was executed
without any additional or negative prompts, and the sampling process was
limited to 20 steps. All experiments were repeated three times under different
random seeds, and results are reported as averages with standard deviations.

###  5.2 Main Results

Table 1: Comparison of test accuracy (%) with state-of-the-art methods on
synthetic datasets CIFAR-10 and CIFAR-100. The best three results are bolded
and the best one is underlined.

|  | Symmetric | Pairflip | Instance
---|---|---|---|---
Datasets | Methods | 30% | 50% | 45% | 30% | 50%
CIFAR-10 | CE | $73.17\pm 1.13$ | $52.59\pm 0.70$ | $51.49\pm 0.42$ | $71.56\pm 0.19$ | $49.20\pm 0.42$
APL | $85.54\pm 0.51$ | $78.36\pm 0.47$ | $80.84\pm 0.72$ | $77.57\pm 0.15$ | $39.45\pm 6.51$
PCE | $86.12\pm 0.85$ | $74.03\pm 4.96$ | $65.08\pm 3.41$ | $85.64\pm 0.72$ | $\bm{64.82\pm 4.13}$
AUL | $88.09\pm 0.78$ | $82.81\pm 1.16$ | $56.80\pm 2.69$ | $86.35\pm 0.90$ | $60.75\pm 3.77$
CELC | $82.51\pm 0.22$ | $\bm{85.08\pm 3.95}$ | $\bm{85.72\pm 4.52}$ | $86.67\pm 1.47$ | $61.85\pm 4.98$
T-Revision | $88.39\pm 0.38$ | $83.40\pm 0.65$ | $83.61\pm 1.06$ | $\bm{89.07\pm 0.35}$ | $\bm{66.93\pm 4.14}$
Identifiability | $87.12\pm 1.69$ | $83.43\pm 2.11$ | $83.65\pm 2.46$ | $80.47\pm 1.54$ | $55.25\pm 3.78$
Joint | $\bm{89.34\pm 0.52}$ | $85.06\pm 0.29$ | $80.52\pm 1.90$ | $\bm{88.41\pm 1.02}$ | $64.12\pm 3.89$
Co-teaching | $88.93\pm 0.56$ | $74.02\pm 0.04$ | $84.19\pm 0.68$ | $87.07\pm 0.35$ | $60.09\pm 3.31$
SIGUA | $83.19\pm 1.26$ | $77.92\pm 3.11$ | $70.39\pm 1.94$ | $82.90\pm 2.00$ | $30.95\pm 9.70$
Co-Dis | $\bm{89.20\pm 0.13}$ | $\bm{85.36\pm 0.94}$ | $\bm{85.02\pm 1.33}$ | $87.13\pm 0.25$ | $62.77\pm 3.90$
Ours | $\underline{\bm{90.98\pm 0.20}}$ | $\underline{\bm{87.95\pm 0.12}}$ | $\underline{\bm{87.42\pm 0.11}}$ |  $\underline{\bm{89.18\pm 0.20}}$ | $\underline{\bm{77.81\pm 0.30}}$
CIFAR-100 | CE | $50.99\pm 1.29$ | $34.5\pm 0.96$ | $37.03\pm 0.41$ | $50.33\pm 2.14$ | $34.70\pm 1.45$
APL | $55.78\pm 0.91$ | $46.96\pm 0.81$ | $49.55\pm 1.05$ | $43.30\pm 1.57$ | $29.01\pm 0.09$
PCE | $58.84\pm 1.32$ | $42.63\pm 2.02$ | $41.05\pm 2.83$ | $55.72\pm 1.96$ | $38.72\pm 3.01$
AUL | $\underline{\bm{69.89\pm 0.21}}$ | $\bm{60.00\pm 0.40}$ | $39.37\pm 1.61$ | $\underline{\bm{67.75\pm 1.84}}$ | $40.27\pm 1.76$
CELC | $\bm{67.96\pm 1.88}$ | $\bm{60.71\pm 2.39}$ | $\bm{52.53\pm 3.17}$ | $\bm{66.25\pm 1.93}$ | $\bm{47.52\pm 3.93}$
T-Revision | $62.97\pm 0.46$ | $43.60\pm 0.94$ | $49.33\pm 1.10$ | $56.46\pm 1.45$ | $40.78\pm 1.75$
Identifiability | $50.53\pm 1.52$ | $34.87\pm 2.36$ | $38.16\pm 2.68$ | $52.48\pm 1.93$ | $36.72\pm 3.10$
Joint | $63.69\pm 0.84$ | $55.62\pm 1.68$ | $49.77\pm 1.15$ | $64.15\pm 1.11$ | $\bm{45.47\pm 2.73}$
Co-teaching | $59.49\pm 0.36$ | $52.19\pm 1.42$ | $47.53\pm 1.39$ | $56.71\pm 1.26$ | $42.09\pm 1.73$
SIGUA | $54.22\pm 0.90$ | $50.64\pm 3.92$ | $39.92\pm 2.33$ | $53.19\pm 2.64$ | $38.50\pm 1.69$
Co-Dis | $64.02\pm 1.37$ | $54.55\pm 2.06$ | $\bm{50.02\pm 2.80}$ | $59.15\pm 1.92$ | $43.38\pm 1.25$
Ours | $\bm{68.16\pm 0.53}$ | $\underline{\bm{60.78\pm 0.46}}$ | $\underline{\bm{60.31\pm 0.37}}$ |  $\bm{65.68\pm 0.48}$ | $\underline{\bm{57.21\pm 0.60}}$

We validated the effectiveness of our method on the CIFAR-10 and CIFAR-100
synthetic datasets. For each dataset, we utilized 90% of the samples with
noisy labels for training and reserved 10% as validation samples, with testing
conducted on clean samples. Part of the baseline results was taken from
previous work (Xia et al., 2023b). As shown in Table 1, our results exhibited
state-of-the-art performance across nearly all conditions on both datasets.
Notably, under more challenging noise conditions, 50% instance-dependent noise
on CIFAR-10 and 45% symmetric noise on CIFAR-100, our method significantly
outperformed the current state-of-the-art baselines, which is mainly
attributed to the robustness of our method to the noise rate. Importantly, our
method demonstrates enhanced performance under higher noise rates, due to
EchoMod’s noise-independence. This attribute allows the model to learn
consistent instance features across different types of noise and noise rates.
The primary cause of performance variation across different settings is the
inherent difference in the number of clean samples within the datasets. Our
approach also exhibited remarkable performance on the real-world dataset,
CIFAR-10N, outperforming the state-of-the-art existing methods in all real
noise conditions. Furthermore, our proposed method demonstrated exceptional
robustness, with standard deviations in performance across all conditions
within floating-point precision.

Table 2: Comparison of test accuracy (%) with state-of-the-art methods on
real-world datasets CIFAR-10N. The best three results are bolded and the best
one is underlined.

Datasets | Methods | Random 1 | Random 2 | Random 3 | Worst
---|---|---|---|---|---
CIFAR-10N | CE | $83.17\pm 0.48$ | $82.74\pm 0.42$ | $82.90\pm 0.28$ | $76.57\pm 0.23$
APL | $84.40\pm 0.26$ | $84.45\pm 0.50$ | $84.35\pm 0.43$ | $78.16\pm 0.17$
PCE | $63.06\pm 0.37$ | $62.26\pm 0.36$ | $35.47\pm 0.36$ | $33.80\pm 0.33$
AUL | $76.26\pm 0.28$ | $75.24\pm 0.20$ | $75.48\pm 0.40$ | $63.61\pm 1.62$
CELC | $83.11\pm 0.14$ | $83.09\pm 0.22$ | $82.60\pm 0.04$ | $73.49\pm 0.50$
T-Revision | $80.99\pm 0.26$ | $78.99\pm 1.59$ | $78.80\pm 1.87$ | $\bm{78.37\pm 0.96}$
Identifiability | $82.52\pm 0.87$ | $81.97\pm 0.85$ | $82.09\pm 0.73$ | $71.62\pm 1.16$
Joint | $\bm{88.20\pm 0.29}$ | $\bm{87.54\pm 0.33}$ | $\bm{87.67\pm 0.22}$ | $\bm{84.29\pm 0.40}$
Co-teaching | $82.28\pm 0.13$ | $82.45\pm 0.23$ | $82.09\pm 0.24$ | $79.62\pm 0.25$
SIGUA | $\bm{87.67\pm 1.18}$ | $\bm{89.01\pm 0.34}$ | $\bm{88.40\pm 0.42}$ | $80.65\pm 1.29$
Co-Dis | $80.81\pm 0.23$ | $80.36\pm 0.20$ | $80.76\pm 0.13$ | $78.12\pm 0.25$
Ours | $\underline{\bm{89.42\pm 0.12}}$ | $\underline{\bm{89.31\pm 0.06}}$ |  $\underline{\bm{89.80\pm 0.25}}$ | $\underline{\bm{84.35\pm 0.09}}$

###  5.3 In-Depth Analyses

Table 3: Comparison with state-of-the-art methods on CIFAR-10 and CIFAR-100 in
accuracy (%).

| CIFAR-10 Pairflip-45% | CIFAR-10 IDN-50% | CIFAR-100 Pairflip-45% | CIFAR-100 IDN-50%
---|---|---|---|---
CE | 51.49 | 49.20 | 37.03 | 34.70
Instance Modification Only | 42.77 | 44.98 | 15.69 | 16.36
EchoSelect Only | 79.46 | 65.77 | 44.24 | 41.24
Ours | 87.42 | 77.81 | 60.31 | 57.21

Ablation Analysis To assess the effectiveness of Instance Modification and
EchoSelect, we conducted ablation studies by systematically disabling these
components. Specifically, we evaluated two configurations: Instance
Modification Only and EchoSelect Only, and compared both against the standard
Cross-Entropy Loss (CE) as a baseline. These experiments were carried out
under several settings with high noise rates, which presented significant
challenges for the model. The experimental results in Table 3 revealed that in
the configuration using only Instance Modification, the model’s accuracy not
only failed to exceed the baseline CE but also showed poorer performance. This
decline primarily stems from the data distribution shift caused by solely
using modified instances, which adversely affects the model’s generalization
capability and performance. In contrast, while using only the EchoSelect
component did improve performance, it still fell short compared to EchoAlign.
This indicates that although EchoSelect significantly reduces the impact of
noise, its effectiveness is limited by the available number of samples.

Table 4: Comparison of sample selection quality under CIFAR-10 instance-dependent noise. Noise rate |  | BLTM | Ours
---|---|---|---
select. acc. | # of selected examples | # of selected examples
IDN-30% | 96% | 17673 / 50000 | 26524 / 50000
99% | 10673 / 50000 | 19010 / 50000
IDN-50% | 94% | 8029 / 50000 | 11660 / 50000
98% | 5098 / 50000 | 6090 / 50000

(a)

(b)

Figure 4: (a) Comparison of the effect of the threshold ($\tau$) on accuracy
at different settings of 30% noise rate. (b) Evaluation of thresholding
effects on the quality and quantity of sample selection under 30% instance-
dependent noise on CIFAR-10.

Sensitivity Analysis The performance of our proposed EchoSelect method is
influenced by the choice of the threshold value $\tau$, which decisively
affects the number and quality of samples selected from noisy datasets.
According to the assumptions of EchoSelect, the optimal threshold value is
theoretically set around 0.5. To validate the efficacy of our method, we used
the state-of-the-art BLTM (Yang et al., 2022) approach as a baseline, with
results directly cited from its original publication. As shown in Table 4, our
EchoSelect was able to select significantly more samples than BLTM under the
same accuracy conditions. Particularly, in environments with 30% instance-
dependent noise, when the accuracy reached 99%, the number of samples selected
by EchoSelect was almost twice that of BLTM. Figure 4(a) clearly demonstrates
that the threshold $\tau$ exhibits robustness and stability across different
classes and types of noise, predominantly influenced by the noise rate. The
slight performance disparity between CIFAR-10 and CIFAR-100 as depicted in the
figure is attributed to CIFAR-100 containing 20 superclasses, with high
similarity among subclasses that increases the complexity of classification.
Furthermore, Figure 4(b) details how adjustments to the threshold value $\tau$
affect the quantity and precision of sample selection. The smooth transitions
displayed, along with a clearly defined optimal equilibrium region, further
affirm the efficacy of our method in various noise environments. To conclude,
the threshold $\tau$ is not sensitive and robust to set.

##  6 Conclusion and Discussion

Conclusion This work provided a positive answer to whether we can effectively
treat label noise as accurate through instance modification. Theoretical
analysis supports the instance modification that this alignment process allows
models to learn meaningful patterns despite the presence of labeling errors.
To overcome the challenges of instance modification, we proposed the EchoAlign
framework. It integrates a controllable generative model with strategic sample
selection to create a robust training dataset. Extensive experiments on
diverse datasets demonstrate the superiority of EchoAlign over existing
methods, particularly in scenarios with high levels of label noise. Future
directions to enhance EchoAlign’s impact include: investigating supervised or
self-supervised extensions and exploring its effectiveness in broader
applications like medical imaging or real-time systems.

Potential Limitations and Social Impact While EchoAlign shows promising
results, its success partially depends on the capabilities of the controllable
generative model. Additionally, the computational cost associated with
processing large datasets through controllable generative models could be a
barrier in resource-limited settings. EchoAlign has the potential to enhance
the robustness and fairness of AI systems in domains where noisy labels are
prevalent, such as with real-world data ranging from medical diagnosis to
social media content analysis. By addressing label noise, EchoAlign could
contribute to fairer AI systems. However, it’s crucial to recognize that
EchoAlign is a tool, and like any tool, its impact depends on how it is used.
Careful consideration is needed in the deployment of AI systems to address
ethical concerns and ensure positive societal outcomes.

## References

  * Achiam et al. (2023) J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ , 2023.
  * Arazo et al. (2019) E. Arazo, D. Ortego, P. Albert, N. O’Connor, and K. McGuinness.  Unsupervised label noise modeling and loss correction.  In _International conference on machine learning_ , pages 312–321. PMLR, 2019.
  * Arpit et al. (2017) D. Arpit, S. Jastrzebski, N. Ballas, D. Krueger, E. Bengio, M. S. Kanwal, T. Maharaj, A. Fischer, A. C. Courville, Y. Bengio, and S. Lacoste-Julien.  A closer look at memorization in deep networks.  In _ICML_ , pages 233–242, 2017.
  * Bai et al. (2021) Y. Bai, E. Yang, B. Han, Y. Yang, J. Li, Y. Mao, G. Niu, and T. Liu.  Understanding and improving early stopping for learning with noisy labels.  In _NeurIPS_ , pages 24392–24403, 2021.
  * Bai et al. (2023) Y. Bai, Z. Han, E. Yang, J. Yu, B. Han, D. Wang, and T. Liu.  Subclass-dominant label noise: A counterexample for the success of early stopping.  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023.
  * Berthon et al. (2021) A. Berthon, B. Han, G. Niu, T. Liu, and M. Sugiyama.  Confidence scores make instance-dependent label-noise learning possible.  In _ICML_ , Proceedings of Machine Learning Research, pages 825–836, 2021.
  * Bose et al. (2022) J. Bose, R. P. Monti, and A. Grover.  Controllable generative modeling via causal reasoning.  _Transactions on Machine Learning Research_ , 2022.
  * Chen et al. (2023a) H. Chen, J. Wang, A. Shah, R. Tao, H. Wei, X. Xie, M. Sugiyama, and B. Raj.  Understanding and mitigating the label noise in pre-training on downstream tasks.  _arXiv preprint arXiv:2309.17002_ , 2023a.
  * Chen et al. (2024) H. Chen, B. Raj, X. Xie, and J. Wang.  On catastrophic inheritance of large foundation models.  _arXiv preprint arXiv:2402.01909_ , 2024.
  * Chen et al. (2023b) J. Chen, R. Zhang, T. Yu, R. Sharma, Z. Xu, T. Sun, and C. Chen.  Label-retrieval-augmented diffusion models for learning from noisy labels.  _ArXiv_ , abs/2305.19518, 2023b.  doi: 10.48550/arXiv.2305.19518.
  * Chen et al. (2023c) T. Chen, Y. Liu, Z. Wang, J. Yuan, Q. You, H. Yang, and M. Zhou.  Improving in-context learning in diffusion models with visual context-modulated prompts.  _arXiv preprint arXiv:2312.01408_ , 2023c.
  * Cheng et al. (2020) J. Cheng, T. Liu, K. Ramamohanarao, and D. Tao.  Learning with bounded instance and label-dependent label noise.  In _International conference on machine learning_ , pages 1789–1799. PMLR, 2020.
  * Dosovitskiy et al. (2021) A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _ICLR_ , 2021.
  * Du et al. (2023) H. Du, H. Yuan, Z. Huang, P. Zhao, and X. Zhou.  Sequential recommendation with diffusion models.  _ArXiv_ , abs/2304.04541, 2023.  doi: 10.48550/arXiv.2304.04541.
  * Franceschi et al. (2023) J.-Y. Franceschi, M. Gartrell, L. D. Santos, T. Issenhuth, E. de B’ezenac, M. Chen, and A. Rakotomamonjy.  Unifying gans and score-based diffusion as generative particle models.  _ArXiv_ , abs/2305.16150, 2023.  doi: 10.48550/arXiv.2305.16150.
  * Goldberger and Ben-Reuven (2016) J. Goldberger and E. Ben-Reuven.  Training deep neural-networks using a noise adaptation layer.  In _International conference on learning representations_ , 2016.
  * Gu et al. (2023) K. Gu, X. Masotto, V. Bachani, B. Lakshminarayanan, J. Nikodem, and D. Yin.  An instance-dependent simulation framework for learning with label noise.  _Machine Learning_ , 112(6):1871–1896, 2023.
  * Han et al. (2018) B. Han, Q. Yao, X. Yu, G. Niu, M. Xu, W. Hu, I. Tsang, and M. Sugiyama.  Co-teaching: Robust training of deep neural networks with extremely noisy labels.  In _NeurIPS_ , pages 8527–8537, 2018.
  * Han et al. (2020) B. Han, G. Niu, X. Yu, Q. Yao, M. Xu, I. Tsang, and M. Sugiyama.  SIGUA: Forgetting may make learning with noisy labels more robust.  In _International Conference on Machine Learning_ , pages 4006–4016, 2020.
  * Han et al. (2022a) Z. Han, X.-J. Gui, H. Sun, Y. Yin, and S. Li.  Towards accurate and robust domain adaptation under multiple noisy environments.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 45(5):6460–6479, 2022a.
  * Han et al. (2022b) Z. Han, H. Sun, and Y. Yin.  Learning transferable parameters for unsupervised domain adaptation.  _IEEE Transactions on Image Processing_ , 31:6424–6439, 2022b.
  * He et al. (2016) K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _CVPR_ , pages 770–778, 2016.
  * Kim et al. (2021) T. Kim, J. Ko, S. Cho, J. Choi, and S. Yun.  FINE samples for learning with noisy labels.  In _NeurIPS_ , pages 24137–24149, 2021.
  * Kingma et al. (2021) D. Kingma, T. Salimans, B. Poole, and J. Ho.  Variational diffusion models.  _Advances in neural information processing systems_ , 34:21696–21707, 2021.
  * Krizhevsky et al. (2009) A. Krizhevsky, G. Hinton, et al.  Learning multiple layers of features from tiny images.  _Technical report_ , 2009.
  * Li et al. (2020) J. Li, R. Socher, and S. C. H. Hoi.  Dividemix: Learning with noisy labels as semi-supervised learning.  In _ICLR_ , 2020.
  * Li et al. (2017) W. Li, L. Wang, W. Li, E. Agustsson, and L. Van Gool.  Webvision database: Visual learning and understanding from web data.  _arXiv preprint arXiv:1708.02862_ , 2017.
  * Liu et al. (2020) S. Liu, J. Niles-Weed, N. Razavian, and C. Fernandez-Granda.  Early-learning regularization prevents memorization of noisy labels.  In _NeurIPS_ , pages 20331–20342, 2020.
  * Liu and Tao (2015) T. Liu and D. Tao.  Classification with noisy labels by importance reweighting.  _IEEE Transactions on pattern analysis and machine intelligence_ , 38(3):447–461, 2015.
  * Liu et al. (2023) Y. Liu, H. Cheng, and K. Zhang.  Identifiability of label noise transition matrix.  In _International Conference on Machine Learning_ , pages 21475–21496. PMLR, 2023.
  * Lu et al. (2022) Y. Lu, Y. Bo, and W. He.  Noise attention learning: Enhancing noise robustness by gradient scaling.  In _NeurIPS_ , 2022.
  * Ma et al. (2020) X. Ma, H. Huang, Y. Wang, S. Romano, S. Erfani, and J. Bailey.  Normalized loss functions for deep learning with noisy labels.  In _ICML_ , 2020.
  * Menon et al. (2018) A. K. Menon, B. Van Rooyen, and N. Natarajan.  Learning from binary labels with instance-dependent noise.  _Machine Learning_ , 107:1561–1595, 2018.
  * Menon et al. (2019) A. K. Menon, A. S. Rawat, S. J. Reddi, and S. Kumar.  Can gradient clipping mitigate label noise?  In _International Conference on Learning Representations_ , 2019.
  * Natarajan et al. (2013) N. Natarajan, I. S. Dhillon, P. K. Ravikumar, and A. Tewari.  Learning with noisy labels.  _Advances in neural information processing systems_ , 26, 2013.
  * Neuberg (2003) L. G. Neuberg.  Causality: models, reasoning, and inference, by judea pearl, cambridge university press, 2000.  _Econometric Theory_ , 19(4):675–685, 2003.
  * Nguyen et al. (2020) D. T. Nguyen, C. K. Mummadi, T. Ngo, T. H. P. Nguyen, L. Beggel, and T. Brox.  SELF: learning to filter noisy labels with self-ensembling.  In _ICLR_ , 2020.
  * Peters et al. (2017) J. Peters, D. Janzing, and B. Schölkopf.  _Elements of causal inference: foundations and learning algorithms_.  The MIT Press, 2017.
  * Radford et al. (2021) A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Reed et al. (2014) S. Reed, H. Lee, D. Anguelov, C. Szegedy, D. Erhan, and A. Rabinovich.  Training deep neural networks on noisy labels with bootstrapping.  _arXiv preprint arXiv:1412.6596_ , 2014.
  * Scott (2015) C. Scott.  A rate of convergence for mixture proportion estimation, with application to learning from noisy labels.  In _Artificial Intelligence and Statistics_ , pages 838–846. PMLR, 2015.
  * Scott et al. (2013) C. Scott, G. Blanchard, and G. Handy.  Classification with asymmetric label noise: Consistency and maximal denoising.  In _Conference on learning theory_ , pages 489–511. PMLR, 2013.
  * Stiennon et al. (2020) N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, and P. F. Christiano.  Learning to summarize with human feedback.  In _NeurIPS_ , pages 3008–3021, 2020.
  * Tan and Le (2019) M. Tan and Q. V. Le.  Efficientnet: Rethinking model scaling for convolutional neural networks.  In _ICML_ , pages 6105–6114, 2019.
  * Tanaka et al. (2018) D. Tanaka, D. Ikami, T. Yamasaki, and K. Aizawa.  Joint optimization framework for learning with noisy labels.  In _CVPR_ , pages 5552–5560, 2018.
  * Team et al. (2023) G. Team, R. Anil, S. Borgeaud, Y. Wu, J.-B. Alayrac, J. Yu, R. Soricut, J. Schalkwyk, A. M. Dai, A. Hauth, et al.  Gemini: a family of highly capable multimodal models.  _arXiv preprint arXiv:2312.11805_ , 2023.
  * Wang et al. (2019) X. Wang, S. Wang, J. Wang, H. Shi, and T. Mei.  Co-mining: Deep face recognition with noisy labels.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 9358–9367, 2019.
  * Wang et al. (2023) Y. Wang, Z. Liu, L. Yang, and P. S. Yu.  Conditional denoising diffusion for sequential recommendation.  _ArXiv_ , abs/2304.11433, 2023.  doi: 10.48550/arXiv.2304.11433.
  * Wei et al. (2023) H. Wei, H. Zhuang, R. Xie, L. Feng, G. Niu, B. An, and Y. Li.  Mitigating memorization of noisy labels by clipping the model prediction.  In _International Conference on Machine Learning_. PMLR, 2023.
  * Wei et al. (2022) J. Wei, Z. Zhu, H. Cheng, T. Liu, G. Niu, and Y. Liu.  Learning with noisy labels revisited: A study using real-world human annotations.  In _International Conference on Learning Representations_ , 2022.
  * Welinder et al. (2010) P. Welinder, S. Branson, S. J. Belongie, and P. Perona.  The multidimensional wisdom of crowds.  In _NeurIPS_ , pages 2424–2432, 2010.
  * Wu et al. (2020) P. Wu, S. Zheng, M. Goswami, D. N. Metaxas, and C. Chen.  A topological filter for learning with label noise.  In _NeurIPS_ , pages 21382–21393, 2020.
  * Xia et al. (2019) X. Xia, T. Liu, N. Wang, B. Han, C. Gong, G. Niu, and M. Sugiyama.  Are anchor points really indispensable in label-noise learning?  In _NeurIPS_ , pages 6835–6846, 2019.
  * Xia et al. (2020) X. Xia, T. Liu, B. Han, N. Wang, M. Gong, H. Liu, G. Niu, D. Tao, and M. Sugiyama.  Part-dependent label noise: Towards instance-dependent label noise.  _Advances in Neural Information Processing Systems_ , 33:7597–7610, 2020.
  * Xia et al. (2021) X. Xia, T. Liu, B. Han, C. Gong, N. Wang, Z. Ge, and Y. Chang.  Robust early-learning: Hindering the memorization of noisy labels.  In _ICLR_ , 2021.
  * Xia et al. (2023a) X. Xia, B. Han, Y. Zhan, J. Yu, M. Gong, C. Gong, and T. Liu.  Combating noisy labels with sample selection by mining high-discrepancy examples.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 1833–1843, 2023a.
  * Xia et al. (2023b) X. Xia, P. Lu, C. Gong, B. Han, J. Yu, and T. Liu.  Regularly truncated m-estimators for learning with noisy labels.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023b.
  * Xiao et al. (2015) T. Xiao, T. Xia, Y. Yang, C. Huang, and X. Wang.  Learning from massive noisy labeled data for image classification.  In _CVPR_ , pages 2691–2699, 2015.
  * Yang et al. (2022) S. Yang, E. Yang, B. Han, Y. Liu, M. Xu, G. Niu, and T. Liu.  Estimating instance-dependent bayes-label transition matrix using a deep neural network.  In _ICML_ , pages 25302–25312, 2022.
  * Yao et al. (2021) Y. Yao, T. Liu, M. Gong, B. Han, G. Niu, and K. Zhang.  Instance-dependent label-noise learning under a structural causal model.  _Advances in Neural Information Processing Systems_ , 34:4409–4420, 2021.
  * Yao et al. (2023a) Y. Yao, M. Gong, Y. Du, J. Yu, B. Han, K. Zhang, and T. Liu.  Which is better for learning with noisy labels: the semi-supervised method or modeling label noise?  In _International Conference on Machine Learning_ , pages 39660–39673. PMLR, 2023a.
  * Yao et al. (2023b) Y. Yao, T. Liu, M. Gong, B. Han, G. Niu, and K. Zhang.  Causality encourages the identifiability of instance-dependent label noise.  In _Machine Learning for Causal Inference_ , pages 247–264. Springer, 2023b.
  * Yu et al. (2018) X. Yu, T. Liu, M. Gong, and D. Tao.  Learning with biased complementary labels.  In _ECCV_ , pages 69–85, 2018.
  * Yu et al. (2019) X. Yu, B. Han, J. Yao, G. Niu, I. Tsang, and M. Sugiyama.  How does disagreement help generalization against label corruption?  In _International Conference on Machine Learning_ , pages 7164–7173. PMLR, 2019.
  * Zhang et al. (2018) H. Zhang, M. Cissé, Y. N. Dauphin, and D. Lopez-Paz.  mixup: Beyond empirical risk minimization.  In _ICLR_ , 2018.
  * Zhang et al. (2017) J. Zhang, V. S. Sheng, T. Li, and X. Wu.  Improving crowdsourced label quality using noise correction.  _IEEE transactions on neural networks and learning systems_ , 29(5):1675–1688, 2017.
  * Zhang et al. (2023) L. Zhang, A. Rao, and M. Agrawala.  Adding conditional control to text-to-image diffusion models.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 3836–3847, 2023.
  * Zhao et al. (2023) S. Zhao, D. Chen, Y.-C. Chen, J. Bao, S. Hao, L. Yuan, and K.-Y. K. Wong.  Uni-controlnet: All-in-one control to text-to-image diffusion models.  _Advances in Neural Information Processing Systems_ , 2023.
  * Zhou et al. (2023) X. Zhou, X. Liu, D. Zhai, J. Jiang, and X. Ji.  Asymmetric loss functions for noise-tolerant learning: Theory and applications.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2023.
  * Zhuang et al. (2023) Y. Zhuang, Y. Yu, L. Kong, X. Chen, and C. Zhang.  Dygen: Learning from noisy labels via dynamics-enhanced generative modeling.  In _Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_ , 2023.  doi: 10.1145/3580305.3599318.

##  Appendix A Proof of the Theorem on the Effectiveness of Instance
Modification

To provide a comprehensive proof of the theorem regarding the effectiveness of
instance modification in learning from noisy labels, we will assume the
definitions and setup described in the theorem’s statement. We will address
each component of the theorem, demonstrating how the instance modification
approach theoretically leads to improvements in alignment, error reduction,
estimation stability, and generalization.

Proof. We prove each component of Theorem 3.1 regarding the effectiveness of
instance modification as follows:

1\. Alignment:

Claim: The mutual information between $X^{\prime}$ and $\tilde{Y}$,
$I(X^{\prime};\tilde{Y})$, is maximized relative to $I(X;\tilde{Y})$,
indicating better alignment of modified instances with their noisy labels.

###### Proof.

Mutual information $I(X;Y)$ measures the amount of information one random
variable contains about another. By modifying $X$ into $X^{\prime}$ using the
transformation $T$, which incorporates $\tilde{Y}$, we embed information about
$\tilde{Y}$ directly into $X^{\prime}$. Since $X^{\prime}$ is derived from
both $X$ and $\tilde{Y}$, it inherently contains all the information $X$ has
about $\tilde{Y}$ and additional information from the direct dependence on
$\tilde{Y}$. Therefore, $I(X^{\prime};\tilde{Y})\geq I(X;\tilde{Y})$. To
provide a more rigorous mathematical proof of this claim, we proceed as
follows:

The mutual information $I(X;\tilde{Y})$ is defined as:

$I(X;\tilde{Y})=H(X)-H(X\mid\tilde{Y})$

Similarly, the mutual information $I(X^{\prime};\tilde{Y})$ is:

$I(X^{\prime};\tilde{Y})=H(X^{\prime})-H(X^{\prime}\mid\tilde{Y})$

Since $X^{\prime}$ is generated by the transformation $T$ which takes $X$ and
$\tilde{Y}$ as inputs, i.e., $X^{\prime}=T(X,\tilde{Y};\theta)$, we can
analyze the changes in entropy and conditional entropy as follows:

$H(X^{\prime}|\tilde{Y})\leq H(X|\tilde{Y})+H(T|\tilde{Y},X)$

Because $T$ is a deterministic transformation, $H(T|\tilde{Y},X)=0$.
Therefore,

$H(X^{\prime}|\tilde{Y})\leq H(X|\tilde{Y})$

Considering that $X^{\prime}$ incorporates information from both $X$ and
$\tilde{Y}$, the entropy of $X^{\prime}$ satisfies:

$H(X^{\prime})\geq H(X)$

Using the definitions of mutual information and the results from the entropy
and conditional entropy comparisons, we get:

$I(X^{\prime};\tilde{Y})=H(X^{\prime})-H(X^{\prime}|\tilde{Y})\geq
H(X)-H(X|\tilde{Y})=I(X;\tilde{Y})$

This inequality shows that the mutual information between $X^{\prime}$ and
$\tilde{Y}$ is greater than or equal to the mutual information between $X$ and
$\tilde{Y}$.

∎

2\. Error Reduction:

Claim: Compared to a model trained on the original instances $X$, the expected
prediction error $\mathbb{E}_{X^{\prime},Y}[(Y-f(X^{\prime}))^{2}]$ is
minimized, where $f$ is the prediction function trained using the modified
instances $X^{\prime}$. This assumes that the distribution of $X^{\prime}$
does not deviate significantly from the distribution of $X$, ensuring that the
learned model generalizes well to the original distribution.

Setup: Let $f_{X}$ denote the prediction model trained on the original
features $X$ and $f_{X^{\prime}}$ the model trained on modified features
$X^{\prime}$. Assume a linear relationship for simplicity, though the model
can be generalized to non-linear relationships.

Formulation:

  * •

The true model: $Y=\bar{f}(X)+\epsilon$, where $\epsilon$ represents random
error with $\mathbb{E}[\epsilon]=0$ and $\text{Var}(\epsilon)=\sigma^{2}$.

  * •

Modified features: $X^{\prime}=T(X,\tilde{Y};\theta)$, where $T$ aims to
optimize alignment with $\tilde{Y}$.

Objective: To prove that
$\mathbb{E}[(Y-f_{X^{\prime}}(X^{\prime}))^{2}]<\mathbb{E}[(Y-f_{X}(X))^{2}]$.

Proof.

  * •

Model Definitions:

    * –

Model for $X$: $f_{X}(X)=\beta_{X}^{T}X$

    * –

Model for $X^{\prime}$:
$f_{X^{\prime}}(X^{\prime})=\beta_{X^{\prime}}^{T}X^{\prime}$

  * •

Error Formulations:

    * –

Error for model using $X$: $\epsilon_{X}=Y-f_{X}(X)=Y-\beta_{X}^{T}X$

    * –

Error for model using $X^{\prime}$:
$\epsilon_{X^{\prime}}=Y-f_{X^{\prime}}(X^{\prime})=Y-\beta_{X^{\prime}}^{T}X^{\prime}$

  * •

Expected Prediction Error:

$\displaystyle\text{For }X$
$\displaystyle:\mathbb{E}[\epsilon_{X}^{2}]=\mathbb{E}[(Y-\beta_{X}^{T}X)^{2}]=\mathbb{E}[Y^{2}]-2\mathbb{E}[Y(\beta^{T}_{X}X)]+\mathbb{E}[(\beta^{T}_{X}X)^{2})]$
$\displaystyle\text{For }X^{\prime}$
$\displaystyle:\mathbb{E}[\epsilon_{X^{\prime}}^{2}]=\mathbb{E}[(Y-\beta_{X^{\prime}}^{T}X^{\prime})^{2}]=\mathbb{E}[Y^{2}]-2\mathbb{E}[Y(\beta^{T}_{X^{\prime}}X^{\prime})]+\mathbb{E}[(\beta^{T}_{X^{\prime}}X^{\prime})^{2})]$

Since $\mathbb{E}[Y^{2}]$ is the same in both cases, we mainly compare
$\mathbb{E}[Y\beta_{X}^{T}X]$ with
$\mathbb{E}[Y\beta_{X^{\prime}}^{T}X^{\prime}]$, and
$\mathbb{E}[(\beta_{X}^{T}X)^{2}]$ with
$\mathbb{E}[(\beta_{X^{\prime}}^{T}X^{\prime})^{2}]$.

  * •

Reduction in Error:

    * –

We assume that $\theta$ in $T$ is conserved, $\beta_{X^{\prime}}$ is chosen
such that the variance of $\epsilon_{X^{\prime}}$ is minimized. This is
achieved by ensuring $\beta_{X^{\prime}}$ aligns with the modified feature
space of $X^{\prime}$ to capture more relevant information about $Y$ contained
implicitly in $\tilde{Y}$.

    * –

The additional alignment information in $X^{\prime}$ compared to $X$ reduces
the component of the error due to noise or irrelevant variability in $X$.
Hence, $\mathbb{E}[\epsilon_{X^{\prime}}^{2}]$ is expected to be less than
$\mathbb{E}[\epsilon_{X}^{2}]$.

  * •

Mathematical Justification:

To rigorously prove this, we use covariance formulas. Assume $\beta_{X}$ and
$\beta_{X^{\prime}}$ are the optimal regression coefficients obtained by
minimizing the mean squared error on $X$ and $X^{\prime}$ respectively:

We want to find $\beta_{X}$ that minimizes the sum of squared errors:

$\min_{\beta_{X}}\mathbb{E}[(Y-\beta^{T}_{X}X)^{2}]=\min_{\beta_{X}}(\mathbb{E}[Y^{2}]-2\mathbb{E}[Y(\beta^{T}_{X}X)]+\mathbb{E}[(\beta^{T}_{X}X)^{2})])$

Taking the derivative with respect to $\beta_{X}$ and setting it to zero:

$\displaystyle\frac{\partial}{\partial\beta_{X}}(\mathbb{E}[Y^{2}]-2\mathbb{E}[Y(\beta^{T}_{X}X)]+\mathbb{E}[(\beta^{T}_{X}X)^{2})])$
$\displaystyle=0$ $\displaystyle
0-2\mathbb{E}[YX]+2\mathbb{E}[XX^{T}]\beta_{X}$ $\displaystyle=0$
$\displaystyle\mathbb{E}[XX^{T}]\beta_{X}$ $\displaystyle=E[YX]$

Assuming $\mathbb{E}[XX^{T}]$ is invertible, we get
$\beta_{X}=(\mathbb{E}[XX^{T}])^{-1}E[YX]$.

For $X^{\prime}$, the derivation is similar, we get
$\beta_{X^{\prime}}=(\mathbb{E}[X^{\prime}X^{\prime T}])^{-1}E[YX^{\prime}]$.

Since $X^{\prime}$ contains more information about $Y$, we can assume
$\mathbb{E}[YX^{\prime}]\geq\mathbb{E}[YX]$. This directly leads to
$\beta^{T}_{X^{\prime}}\mathbb{E}[YX^{\prime}]\geq\beta^{T}_{X}\mathbb{E}[YX]$.

Combining these with the expected prediction error formulas:

$\displaystyle\mathbb{E}[(Y-\beta_{X}^{T}X)^{2}]$
$\displaystyle=\mathbb{E}[Y^{2}]-2\beta_{X}^{T}\mathbb{E}[YX]+\beta_{X}^{T}\mathbb{E}[XX^{T}]\beta_{X}$
$\displaystyle\mathbb{E}[(Y-\beta_{X^{\prime}}^{T}X^{\prime})^{2}]$
$\displaystyle=\mathbb{E}[Y^{2}]-2\beta_{X^{\prime}}^{T}\mathbb{E}[YX^{\prime}]+\beta_{X^{\prime}}^{T}\mathbb{E}[X^{\prime}X^{\prime
T}]\beta_{X^{\prime}}$

Because $X^{\prime}$ aligns better with the information in $\tilde{Y}$, we can
assume
$\mathbb{E}[(\beta_{X^{\prime}}^{T}X^{\prime})^{2}]\leq\mathbb{E}[(\beta_{X}^{T}X)^{2}]$.
Since $\beta_{X^{\prime}}$ is obtained by minimizing
$\mathbb{E}[(Y-\beta_{X^{\prime}}^{T}X^{\prime})^{2}]$, and
$\mathbb{E}[YX^{\prime}]\geq\mathbb{E}[YX]$, it follows that:

$\mathbb{E}[(Y-\beta_{X^{\prime}}^{T}X^{\prime})^{2}]\leq\mathbb{E}[(Y-\beta_{X}^{T}X)^{2}]$

∎

3\. Estimation Stability:

Claim: The variance of the estimator $f$ is reduced when using $X^{\prime}$
compared to $X$, resulting in more stable predictions.

Formulation: Assume the following linear regression models for simplicity,
though the concepts generalize to non-linear models:

  * •

Model using $X$: $f_{X}=\beta_{X}^{T}X+\epsilon_{X}$, where $\epsilon_{X}$ is
the noise term.

  * •

Model using $X^{\prime}$:
$f_{X^{\prime}}=\beta_{X^{\prime}}^{T}X^{\prime}+\epsilon_{X^{\prime}}$, where
$\epsilon_{X^{\prime}}$ is the noise term for the modified model.

Objective: To demonstrate that the variance of the estimator $f_{X^{\prime}}$
is lower than that of $f_{X}$.

Proof.

  * •

Model Definitions and Assumptions: Assume that both $\beta_{X}$ and
$\beta_{X^{\prime}}$ are obtained by ordinary least squares (OLS), implying
that they minimize the respective mean squared errors. The variance of the
estimator in OLS is inversely proportional to the Fisher information of the
model, Fisher information matrix $I(\beta)$ is represented as $X^{T}X$ and
$X^{\prime T}X^{\prime}$, reflecting the variability of input features.

  * •

Variance of Estimators: The variance of each estimator under OLS can be
expressed as follows:

$\displaystyle\text{Var}(f_{X})$ $\displaystyle=\sigma^{2}(X^{T}X)^{-1}$
$\displaystyle\text{Var}(f_{X^{\prime}})$ $\displaystyle=\sigma^{2}(X^{\prime
T}X^{\prime})^{-1}$

where $\sigma^{2}$ is the variance of the error terms $\epsilon_{X}$ and
$\epsilon_{X^{\prime}}$, assumed equal for simplicity.

  * •

Comparative Analysis of Variance: Since $X^{\prime}$ is designed to be more
informative and aligned with $\tilde{Y}$, it is reasonable to assume that
$X^{\prime}$ exhibits higher effective variability in the dimensions that are
most relevant for predicting $Y$. This increased effective variability in
relevant dimensions implies that the matrix $X^{\prime T}X^{\prime}$ is more
well-conditioned (i.e., has larger eigenvalues and a higher determinant) than
$X^{T}X$, leading to a smaller value for its inverse. Specifically, the
increase in eigenvalues indicates an increase in information in relevant
dimensions, making $(X^{\prime T}X^{\prime})^{-1}$ smaller than
$(X^{T}X)^{-1}$ :

$(X^{\prime T}X^{\prime})^{-1}<(X^{T}X)^{-1}$

Hence, $\text{Var}(f_{X^{\prime}})<\text{Var}(f_{X})$.

  * •

Estimation Stability: The reduction in variance implies that $f_{X^{\prime}}$
offers more stable and reliable predictions compared to $f_{X}$. This
stability is crucial when the model is applied in practice, particularly in
the presence of noisy data conditions.

This detailed proof shows that by focusing on feature dimensions that are more
predictive of $Y$, instance modification via $X^{\prime}$ not only improves
the alignment with the noisy labels but also enhances the stability of the
model’s predictions.

∎

4\. Generalization:

Claim: Modifications in $X^{\prime}$ lead to better generalization. By
transforming the original instances to better align with their noisy labels,
the model trained on $X^{\prime}$ is less to overfit to the noise and more
capable of capturing the true underlying patterns in the data.

Setup: Let

  * •

$X$ denotes the original feature space and $X^{\prime}=T(X,\tilde{Y},\theta)$
denote the modified feature space, where $T$ is a transformation, and $\theta$
is fixed, that optimizes some aspect of the data to better align with noisy
labels $\tilde{Y}$.

  * •

$\mathcal{F}$ be the class of functions $f:\mathcal{X}\to\mathbb{R}$
considered by the learning algorithm, where $\mathcal{X}$ is either the space
of $X$ or $X^{\prime}$.

Rademacher Complexity: Rademacher complexity measures the ability of a
function class to fit random noise. The Rademacher complexity for the class of
functions $\mathcal{F}$ applied to the original features $X$ and the modified
features $X^{\prime}$ are defined respectively as:

$\displaystyle\mathfrak{R}_{n}(\mathcal{F}_{X})$
$\displaystyle=\mathbb{E}_{\sigma,X}\left[\sup_{f\in\mathcal{F}_{X}}\frac{1}{n}\sum_{i=1}^{n}\sigma_{i}f(X_{i})\right]$
$\displaystyle\mathfrak{R}_{n}(\mathcal{F}_{X^{\prime}})$
$\displaystyle=\mathbb{E}_{\sigma,X^{\prime}}\left[\sup_{f\in\mathcal{F}_{X^{\prime}}}\frac{1}{n}\sum_{i=1}^{n}\sigma_{i}f(X^{\prime}_{i})\right]$

Generalization Bounds: Using these definitions, the generalization bounds for
binary classification under the 0-1 loss can be expressed for both feature
sets. Assuming the same hypothesis class $\mathcal{F}$, the bounds are:

$\displaystyle\Pr\left(\sup_{f\in\mathcal{F}_{X}}\left|\mathbb{E}[l(f(X),Y)]-\frac{1}{n}\sum_{i=1}^{n}l(f(X_{i}),Y_{i})\right|>\epsilon\right)$
$\displaystyle\leq
2\exp\left(-\frac{2\epsilon^{2}n}{\mathfrak{R}_{n}(\mathcal{F}_{X})^{2}}\right)$
$\displaystyle\Pr\left(\sup_{f\in\mathcal{F}_{X^{\prime}}}\left|\mathbb{E}[l(f(X^{\prime}),Y)]-\frac{1}{n}\sum_{i=1}^{n}l(f(X^{\prime}_{i}),Y_{i})\right|>\epsilon\right)$
$\displaystyle\leq
2\exp\left(-\frac{2\epsilon^{2}n}{\mathfrak{R}_{n}(\mathcal{F}_{X^{\prime}})^{2}}\right)$

Impact of Instance Modification on Feature Space: The transformation $T$ is
designed to adjust features in $X$ to more effectively align with $\tilde{Y}$,
potentially reducing the variability of $X$ that is irrelevant to predicting
$Y$. This transformation can:

  * •

Increase the signal-to-noise ratio in $X^{\prime}$ compared to $X$.

  * •

Focus the variability in $X^{\prime}$ on aspects that are more predictive of
$Y$, based on the information contained in $\tilde{Y}$.

###### Proof.

To show that
$\mathfrak{R}_{n}(\mathcal{F}_{X^{\prime}})<\mathfrak{R}_{n}(\mathcal{F}_{X})$,
we analyze how the transformation $T$ affects the ability of function class
$\mathcal{F}$ to fit random noise:

  * •

Reduction in Effective Variance: Since $T$ reduces the non-predictive
variability of $X$, the effective variance of $X^{\prime}$ within the context
of the function class $\mathcal{F}$ is lower. Lower variance in the features
directly translates to a reduced ability to fit arbitrary labels (noise),
effectively decreasing the Rademacher complexity.

  * •

Alignment with Output: If $T$ enhances the alignment of $X^{\prime}$ with the
output $\tilde{Y}$, the predictions by $\mathcal{F}$ on $X^{\prime}$ are less
variable for random labeling. This reduction in fitting capacity to random
labels also points to a reduced Rademacher complexity.

Consider the expectation of the supremum of correlation with Rademacher
sequences:

$\mathfrak{R}_{n}(\mathcal{F}_{X^{\prime}})=\mathbb{E}_{\sigma,X^{\prime}}\left[\sup_{f\in\mathcal{F}}\frac{1}{n}\sum_{i=1}^{n}\sigma_{i}f(X^{\prime}_{i})\right]<\mathbb{E}_{\sigma,X}\left[\sup_{f\in\mathcal{F}}\frac{1}{n}\sum_{i=1}^{n}\sigma_{i}f(X_{i})\right]=\mathfrak{R}_{n}(\mathcal{F}_{X})$

due to the focused and reduced variability in $X^{\prime}$ relevant to the
output $\tilde{Y}$.

The inequality derived from comparing the Rademacher complexities and the
corresponding generalization bounds provides a theoretical basis for asserting
that instance modification enhances the model’s ability to generalize. This
proof underscores the importance of feature alignment and relevance in
improving machine learning model performance in noisy settings.

∎

##  Appendix B More Details of Theorem and Analysis of Instance Modification

To validate the correctness of our proposed theorem 3.1, we undertook specific
experiments to demonstrate its efficacy. The theorem posits that by applying
an appropriate transformation $T$, the alignment between the instances $X$ and
the noisy labels $\tilde{Y}$ can be optimized, thereby increasing their mutual
information. On the CIFAR-10 dataset, we calculated the mutual information
between 50,000 images and their labels. As observed in Figure 5(a), the mutual
information $I(X^{\prime};\tilde{Y})$ between the transformed instances
$X^{\prime}$ and the noisy label $\tilde{Y}$ is significantly higher than the
mutual information $I(X;\tilde{Y})$ between the original instances $X$ and
$\tilde{Y}$. Figure 5(b) also argues the third point of our theorem, i.e., the
estimator trained on X’ has a better variance than that trained on X, which
illustrates the higher stability and robustness of our method. Furthermore,
concerning prediction error, Figure 6 displays the training and testing
results under different noise types. The results show that both in the
training and testing sets, using the modified samples resulted in
significantly lower errors.

(a)

(b)

Figure 5: (a) illustrates the mutual information between the labels of 50,000
original samples and their corresponding 50,000 modified samples under 50%
instance-dependent noise on CIFAR-10. (b) shows the distribution of the
predictive probability of the estimator $f$ using $X^{\prime}$ and $X$.

(a) 50% symmetric noise

(b) 45% pairflip noise

(c) 50% instance-dependent noise

Figure 6: Figures (a), (b), and (c) respectively illustrate the differences in
training and testing losses between EchoAlign and the CE model under 50%
symmetric noise, 45% pairflip noise, and 50% instance-dependent noise
conditions on the CIFAR-10. The bright peach red and deep burgundy lines
represent the performance of CE and EchoAlign on the test set, respectively,
while the light purple and light coral pink lines denote their performance on
the training set.

##  Appendix C Data preprocessing and hyperparameter settings

Data preprocessing For all datasets, including CIFAR-10, CIFAR-100, and
CIFAR-10N, we adopted a unified and straightforward data augmentation method.
Specifically, we first applied a 4-pixel padding to the images, followed by
random cropping to $32\times 32$ pixels. Subsequently, we applied random
horizontal flipping and normalization.

Hyperparameter settings For the ControlNet controllable generation model, we
used the simplest Canny preprocessor with both the low threshold and high
threshold set to 75. For the prompt, we used the template “a photo of label”
without any additional prompts or negative prompts. The feature maps output by
the preprocessor and the generated images both have a medium size of
$512\times 512$ pixels. The diffusion process uses 20 steps. For the
EchoSelect section, the default threshold is set to 0.4 for all cases with 30%
noise, and 0.52 for cases with 45% and 50% noise. The hyperparameters for the
training part are detailed in Table 5.

Table 5: Training hyperparameters for CIFAR-10/CIFAR-10N and CIFAR-100. | CIFAR-10/CIFAR-10N | CIFAR-100
---|---|---
architecture | ResNet-18 | ResNet-34
optimizer | SGD | SGD
loss function | CE | CE
learning rate(lr) | 0.1 | 0.1
lr decay | 100th and 150th | 100th and 150th
weight decay | $10^{-4}$ | $10^{-4}$
momentum | 0.9 | 0.9
batch size | 128 | 128
training samples | 45,000 | 45,000
training epochs | 200 | 200

##  Appendix D Runtime Analysis

Table 6: Comparison of runtime at different settings using an NVIDIA V100-SXM2. Image resolution | CIFAR-10 | Clothing1M
---|---|---
batch size-1 | batch size-8 | batch size-16 | batch size-16
$256\times 256$ | 31.5 | 5.5 | 4.5 | 129.5
$512\times 512$ | 35.1 | 18.5 | 17.2 | 504.5
$768\times 768$ | 68.5 | 51.9 | ~~~~| ~~~~
Table 7: Comparison of runtime at different computing performance. GPU | CIFAR-10 | Clothing1M
---|---|---
V100-SXM2 | 18.5 | 504.5
RTX 4090 | 8.5 | 338.2

The efficiency of EchoMod is significantly influenced by several factors,
including the choice of the controllable generative model, the GPU’s floating-
point operations per second (FLOPS), the resolution of generated images, batch
size, GPU memory capacity, floating-point precision, and the number of
diffusion steps if a diffusion model is used. In this study, we employ an
NVIDIA V100-SXM2 with 32GB of VRAM, using ControlNet as the benchmark
generative model, and apply mixed precision to assess the impacts of image
size and batch size on runtime. Runtime is measured in GPU hours, which
quantifies the time consumed to perform computational tasks on a single GPU.
As the number of GPUs increases, we observe a super-linear decrease in
runtime. Our experiments are conducted on the CIFAR-10 dataset, and we also
estimate the runtime for processing the Clothing1M dataset on the same GPU
configuration. Table 6 demonstrates that increasing the batch size and
reducing the image resolution both significantly impact runtime. We did not
conduct tests with image resolution at $768\times 768$ and a batch size of 16
due to similar VRAM constraints. Additionally, in Table 7, we compare the
effects of different computing performance on runtime. We conducted tests on
two different GPUs with an image resolution of $512\times 512$ and a batch
size of 8. The NVIDIA V100-SXM2-32GB offers a half-precision compute
capability of 125 Tensor TFLOPS and a single-precision capability of 15.7
TFLOPS. In contrast, the more powerful NVIDIA RTX 4090-24GB GPU provides 165.2
Tensor TFLOPS in half-precision and 82.58 TFLOPS in single-precision.

##  Appendix E Challenges of Instance Modification

Instance modification presents several significant challenges that need to be
addressed to ensure the effectiveness and reliability of the modified
instances. Instance modification primarily relies on Controllable Generative
Models (CGM), which can generate instances that meet specific requirements
under certain conditions. However, Characteristic Shift is a major challenge,
as illustrated in Figure 7(a). Characteristic Shift occurs during instance
modification when the fundamental attributes of the instances are altered.
These changes can lead to inconsistencies and distortions in the original
data, the effect majorly due to the gap between the intrinsic features of the
original instances and the noise labels. For example, if the CGMs
significantly alter key features such as shape or texture during modification,
the models may fail to learn consistent representations. Therefore, addressing
the Characteristic Shift between original instances and noise labels is
crucial. Another major challenge is Distribution Shift, as shown in Figures
7(b) and 7(c). This challenge arises when the statistical distribution of the
modified instances deviates from the original data distribution. Such shifts
can lead to models performing well on modified data but poorly on real-world
data. The T-SNE visualization of CIFAR-10 instance representations shows the
changes in instance distribution before and after using EchoAlign, showcasing
the effectiveness of our method.

(a) Characteristic Shift.

(b) T-SNE Visualization of CIFAR-10 instance representations by using $X$.

(c) T-SNE Visualization of CIFAR-10 instance representations by using
$X^{\prime}$

Figure 7: (Top) Main challenge 1: Characteristic Shift. (Bottom) Main
challenge 2: Distribution Shift.

##  Appendix F More Generation Examples

In this section, we present the results of several publicly available
controllable (CGM) and non-controllable (NGM) generative models that can
easily generate demos. We used “a photo of noisy label” as the prompt and the
original instance as the control condition. As shown in Table 8, for
controllable generative models, we present the results of ControlNet and
UniControl [Zhao et al., 2023]. For non-controllable generative models, we
showcase the performance of GPT-4 [Achiam et al., 2023] and Gemini [Team et
al., 2023].

Table 8: Results of Controllable and Non-Controllable Generative Models |  | CGM | NGM
---|---|---|---
Noisy Label | Original Instance | ControlNet | UniControl | GPT-4 | Gemini
Cat |  |  |  |  |
Magpie |  |  |  |  |
T-shirt |  |  |  |  |
Fabric bag |  |  |  |  |
Dress shoes |  |  |  |  |

##  Appendix G Framework

Figure 8 illustrates the framework of EchoAlign, which comprises two main
modules: EchoMod and EchoSelect. The EchoMod module uses Controllable
Generative Models (CGM) to modify noisy instances, generating instances
consistent with their labels. Specifically, the CGM receives noisy instances
and generates new instances based on the images’ labels and intrinsic
features. Through the EchoMod module, we obtain a set of modified instances.
Subsequently, these modified instances enter the EchoSelect module. In this
module, the system uses cosine similarity evaluation to select the instances
from the noisy data that are most similar to the modified instances. The
similarity evaluation assumes that the features of clean instances are highly
consistent with those of the modified instances, thereby selecting
sufficiently clean samples to enhance the model’s generalization capability.
Finally, the selected clean instances and the modified instances together form
the training instance set used for subsequent model training. In this way, the
EchoAlign framework effectively addresses the issues of characteristic shift
and distribution shift in instance modification through its robust instance
modification and selection processes, ensuring the model’s generalization
ability and robustness.

Figure 8: The framework of EchoAlign.