Collaboratively Learning Federated Models from Noisy Decentralized Data
Haoyuan Li Mathias Funk Nezihe Merve Gürel Aaqib Saeed



























Collaboratively Learning Federated Models from Noisy Decentralized Data
    
    Haoyuan Li1, Mathias Funk1, Nezihe Merve Gürel2, Aaqib Saeed1
    1Eindhoven University of Technology, Eindhoven, Netherlands
    2Delft University of Technology, Delft, Netherlands
    {h.y.li, m.funk, a.saeed}@tue.nl, n.m.gurel@tudelft.nl

    September 9, 2024
==========================================================================================================================================================================================================================================================



§ ABSTRACT 
Federated learning (FL) has emerged as a prominent method for collaboratively training machine learning models using local data from edge devices, all while keeping data decentralized. However, accounting for the quality of data contributed by local clients remains a critical challenge in FL, as local data are often susceptible to corruption by various forms of noise and perturbations, which compromise the aggregation process and lead to a subpar global model. In this work, we focus on addressing the problem of noisy data in the input space, an under-explored area compared to the label noise. We propose a comprehensive assessment of client input in the gradient space, inspired by the distinct disparity observed between the density of gradient norm distributions of models trained on noisy and clean input data. Based on this observation, we introduce a straightforward yet effective approach to identify clients with low-quality data at the initial stage of FL. Furthermore, we propose a noise-aware FL aggregation method, namely Federated Noise-Sifting (), which can be used as a plug-in approach in conjunction with widely used FL strategies. Our extensive evaluation on diverse benchmark datasets under different federated settings demonstrates the efficacy of . Our method effortlessly integrates with existing FL strategies, enhancing the global model's performance by up to 13.68% in IID and 15.85% in non-IID settings when learning from noisy decentralized data.



federated learning, decentralized AI, data quality, data-centric machine learning




§ INTRODUCTION


Recent advances in machine learning (ML) have led to a surge in data generated by edge devices. Nonetheless, the ubiquity and heterogeneity of data across these devices pose a significant challenge to the efficacy of training generalizable models. 

In response to this pressing need, federated learning (FL) is gaining traction as a decentralized paradigm in the field of distributed ML. In FL, the model is learned in a decentralized manner where each client device collaboratively trains a global model without directly sharing the local data with the centralized server <cit.>. 


Despite this, one of the primary challenges in ML is data quality, which directly impacts the performance and reliability of ML models. The complexity and nature of data in the context of deep models, which require large-scale data, significantly amplify these challenges <cit.>. The multifaceted nature of data quality can be compromised in both the input and feature space, leading to issues such as data incompleteness, feature corruption, and label inconsistency <cit.>. 

Notably, the issue of maintaining high-quality data becomes particularly challenging in FL due to its decentralized nature. FL allows to mitigate domain-specific concerns such as IP security, by imposing the requirement that the server has no visibility into client data, thereby increasing the difficulty of ensuring data quality. Moreover, FL is generally vulnerable to (often non-malicious) client failures, especially when it comes to the noisy model update, which can be seen as a special case of data perturbation <cit.>. These failures typically occur on the client side, where the server has no control over. Specifically, unreliable clients unintentionally distort the generalization of the global model by learning from noisy samples or labels with no control over the data collection phase (e.g., due to no direct interest by the federated client). In practice, the utilization of edge devices for FL is a double-edged sword where a large amount of data can be harvested to learn the model, but such data also often entails significant noise contamination. Particularly for tasks of interest like object detection, data collected from image sensors are susceptible to visual distortions, often attributed to the client's lack of technical expertise or environmental interference <cit.>. 





In this work, we mainly consider data corruption in the input space, where external distortions could pollute the sample of client data. While the impact of data corruption in the label space has been well-studied, little attention has been paid to the effects of corruption in the input space, which constitutes the main focus of our study. 

To address the issue of input space corruption, we perform a comprehensive assessment of client input in the gradient space to monitor the presence of noisy client data.
From the density of gradient norm distribution, we observe a distinct disparity between models trained on noisy and clean input data. Inspired by the observation made on the gradient of loss for each client data, we provide a straightforward yet effective approach to identifying clients with low-quality data at the first stage (or training round) in FL. We further propose a noise-aware FL aggregation method, namely Federated Noise Sifting (). It can be simply used as a plug-in approach in conjunction with widely used FL strategy. Specifically, it re-weights the contributions of client models based on the characteristics of data quality (i.e., clean or noisy input) in aggregating a global model (see Figure <ref>). To demonstrate the efficacy of , we perform extensive experiments on diverse benchmark datasets under different heterogeneous settings. Our findings show that our proposed method effortlessly integrates and works well with existing FL aggregation strategies, such as FedAvg <cit.>, FedProx <cit.>, FedTrimmedAvg <cit.>, and FedNova <cit.> which makes it widely applicable.



In this work, we mainly focus on the problem of noisy data in the input space, where the features in the client data are (non-maliciously) corrupted. 


To address this, we first investigate the gradient of the loss function, aiming to detect the presence of noisy client data. Our motivation stems from the fact that the gradient space of the model provides an informative signal regarding the usefulness of the data <cit.> as it directly captures joint information between representations and output <cit.>. Inspired by this finding, we pose the question does the gradient provide a meaningful way to distinguish between models of noisy and clean federated clients? 
From the density of the gradient norm, we see a distinct disparity between models trained on noisy and clean input data. We thus exploit the richness of gradient space to propose a straightforward yet effective approach to identify clients with low-quality data at the beginning stage of FL. We further propose a noise-aware FL aggregation method, namely Federated Noise Sifting (). 
Specifically, it re-weights the contributions of client models based on the data quality (i.e., clean or noisy input) while aggregating a global model (see Figure <ref>). 
can be easily used as a plug-in approach in conjunction with widely used FL strategies. 
To demonstrate the efficacy of , we perform extensive experiments on diverse benchmark datasets under different heterogeneous settings. 
Our findings show that our proposed method effortlessly integrates and works well with existing FL aggregation strategies, such as FedAvg <cit.>, FedProx <cit.>, FedTrimmedAvg <cit.>, and FedNova <cit.> which makes it widely applicable.

Main contributions: Our work makes the following key contributions: 

    
  * Novel single-interaction approach for identifying noisy clients. We propose an approach that employs gradient norm analysis to identify and categorize noisy (low-quality model updates) clients at the initial training stage (i.e., a first-round) in a federated context without revealing individual client data for inspection.
    
  * Robust and plug-in weights aggregation strategy. We introduce a robust and plug-in weights aggregation strategy to mitigate the adverse impacts of noisy clients, named . It offers a versatile and easily integrable solution to enhance the global model's generalization. We further substantiate the effectiveness of  through its application to corrupted client data across six datasets and four FL strategies.
    
  * Systematic conceptualization of noisy FL and benchmark. We present a concept of noisy FL to understand non-malicious client-side data corruption. Based on this concept, we build[we will release the code upon acceptance.] the noisy datasets benchmark specifically tailored for FL environments with noisy input.




§ RELATED WORK


Dealing with data quality issues is a well-studied problem in machine learning (ML). Significant efforts have been devoted to mitigating the adverse effect of low-quality data on the generalization capabilities of ML models. <cit.>, <cit.>, and <cit.> reveal that while Deep Neural Networks (DNNs) exhibit advanced generalization abilities but are notably susceptible to low-quality data, particularly to distortions like noise and blur considerably diminishing their performance. To tackle this challenge, previous work enhances DNN robustness to image distortions by selectively correcting the activations of the most noise-susceptible convolutional filters <cit.>. Likewise, <cit.> treat image distortion identification as multi-label learning and train a multi-task DNN model to improve the model generalization. Instead of reformulating the architecture of the DNN to improve the robustness of the model, <cit.> proposes to stabilize the training progress against the input perturbations. In a complementary study, <cit.> shows that fine-tuning and re-training the model using noisy data can effectively alleviate the effect of image distortion. In addition to this, numerous studies have concentrated on addressing label noise, a form of low-quality data in the label space. Training DNNs on noisy labels tends to overfit the noise patterns, which causes degradation in generalization performance on unseen data.  Most of the existing methods solve the representation learning with noisy labels by estimating the noise transition matrix and innovatively combining methods like adaptation layers, loss corrections, and regularization techniques <cit.>. While these approaches yield substantial results in centralized learning, their applicability is notably limited in federated learning (FL) scenarios, where data is distributed across myriad devices, and the data quality from each client can not be guaranteed due to its private nature.

Recent works in FL mainly focus on addressing the data quality issues pertaining to the label space. Many strategies have been developed to deal with the disparity of label quality in FL. These methods mitigate the impact of label noise by conducting either client selection to re-weight model updates <cit.> or data sampling for label correction or exclusion <cit.>. Additionally, other works tackle this challenge by correcting the label error. <cit.> perform label noise correction using consensus-derived class-wise information for dynamic noise identification and label correction. <cit.> tackles label noise in federated learning by updating local models with globally aligned centroids and correcting labels through global model insights. In spite of the progress that has been made in resolving label noise, data subset selection <cit.>, data valuation <cit.>, dealing with low-quality data in input space (i.e., when noise is in the samples like in images) in the federated setting still remains unexplored. Furthermore, we recognize that client data may be susceptible to backdoor attacks via adversarial methods during both inference and training time <cit.>; these concerns fall outside the scope of our investigation. To this end, our method aims for a flexible and efficient solution suitable for a range of FL strategies to learn robust models using decentralized data.





§ METHOD

We introduce our robust federated learning (FL) method, named Federated Noise Sifting (), designed for learning from noisy input. We begin with defining the challenge of FL in the context of input noise and then describe our single-interaction approach to discover and classify noisy and clean clients. Then, we propose a generic noise-aware aggregation method that can be paired with any federated strategy to mitigate the impact of contributions from noisy clients.



 §.§ Problem Definition


In a typical FL scenario, a total of K clients collaboratively train a machine learning model by optimizing a global objective function <cit.>.
To formulate this setting, consider K disjoint local datasets by clients, where each client indexed by k holds a private dataset D_k = {(x_i, y_i)}_i=1^|D_k|, with each pair (x_i, y_i) representing a data sample such that x_i ∈𝒳 as the input and y_i ∈𝒴 as the corresponding label. 

 The objective of the clients is to minimize the loss ℒ(θ) aggregated from the individual loss of each local client ℓ(θ, D_k) with respect to the aggregated server model parameters θ:

    !min_θℒ(θ)    = min_θ1/∑_k ∈ [K]|D_k|∑_k ∈ [K]∑_(x_i, y_i) ∈ D_kℓ(θ; {(x_i, y_i)}) 
       = min_θ∑_k=1^K w_k ℓ(θ; D_k)

where 
w_k denotes the weight assigned to the k-th client's loss function. Here, we take w_k proportional to the size of local data D_k of client k such that w_k = |D_k| / ∑_k=1^K |D_k|, similar to that of <cit.>.


Noisy Federated Learning    We focus on a noisy federated learning setting where perturbations are present in the input space 𝒳, that often occur due to the unexpected variability in data collection or physical conditions and other environmental factors <cit.>. 

Consider an FL setup consisting of K participating clients, where M out of K clients' local datasets are partially contaminated by some noise, whose function is defined as η(·): 𝒳↦𝒳. 
Conversely, the local datasets of the remaining N clients, where N=K-M, are unaffected.
To simplify the exposure, we index the noisy clients by m and clean clients by n, and denote the respective client data by D_m and D_n, where {∪_m D_m}∪{∪_n D_n}= ∪_k D_k. 
We assume that the local dataset of each noisy client m can be further split disjointly into clean data D_m^clean and noisy data D_m^noisy such that D_m^clean∪ D_m^clean = D_m and D_m^clean∩ D_m^clean = ∅.  

In such noisy federated learning setting, the global objective can be expressed as:

    min_θℒ(θ)    = min_θ∑_n ∈ [K]
    s.t.  D_n  is cleanw_nℓ(θ; D_n)+∑_m ∈ [K]
    s.t.  D_m  is noisyw_mℓ(θ; D_m)
       = min_θ∑_n ∈ [K]
    s.t.  D_n  is cleanw_nℓ(θ; D_n)
         +∑_m ∈ [K]
    s.t.  D_m  is noisyw_m ( |D_m^clean|/|D_m|ℓ(θ; D_m^clean).
         .+|D_m^noisy|/|D_m|ℓ(θ; D_m^noisy) ).

which is aggregated from both noisy clients ℓ_m(θ) and clean clients ℓ(θ)_n respectively. The loss function of noisy clients can be further written as:

    ℓ(θ) = ∑_m=1^Mw_m ( ∑_i=1^|D_m^clean|ℓ(η(x_i), y_i; θ) + ∑_j=1^|D_m^noisy|ℓ(x_j, y_j; θ) )



We assume that the local dataset of each noisy client m can be further split disjointly into clean data D_m^clean={(x_i, y_i)}_i=1^N_m^clean and noisy data D_m^noisy = {(η(x_j), y_j)}_j=1^N_m^noisy. N_m^noisy and N_m^clean respectively correspond to the number of noisy data samples and clean data samples. 

In such noisy federated learning setting, the global objective function can be expressed as:

    min_θℒ(θ) = ∑_n=1^Nw_nℓ_n(θ) + ∑_m=1^Mw_mℓ_m(θ)

which is aggregated from both noisy clients ℓ(θ) and clean clients ℓ(θ) respectively. The loss function of noisy clients can be further rewritten as:

    min_θℓ(θ) = ∑_m=1^Mw_m ( ∑_i=1^|D_m^clean|ℓ(η(x_i), y_i; θ) + ∑_j=1^|D_m^noisy|ℓ(x_j, y_j; θ) )












Noisy Input Data   In decentralized data collection, input corruption typically occurs on the client-side, influenced by varying degrees of real-world noise and varying degrees of corruption from weak to strong. To address this, we propose a definition for noisy FL data that accounts for the intensity and level of noise in each client's local datasets.



Consider the partition of a global training dataset D into K local datasets D_k for k∈[K]. This partitioning follows either an Independent and Identically Distributed (IID) or a non-IID setting <cit.>. Within this set of K clients, we randomly pick up M clients as the noisy clients, in which the local datasets of these clients are contaminated by input noise, such as Gaussian blur, contrast, defocus blur (see the extended list in the Section <ref>). Consequently, the local dataset of m-th noisy client D_m is transformed into noisy local data by applying a randomly selected transformation τ from all the data transformation 𝐓 with a specific severity level ξ∈{low, medium, high}. This can be expressed as:

    D_m^noisy = { η(x, τ, ξ) | x ∈ D_m, τ∈𝐓}.

Thus, we define the noise level of client m as NL_m = |D_m^noisy|/|D_m|, which represents the fraction of noisy corrupted data samples over the entire local data of client m. We further introduce image noise severity to quantify the extent of distortion in digital images. Severity levels are scaled from low to high, indicating the intensity of the noise. An illustration of different levels of image noise severity is shown in Figure <ref>.

In addition to the image distortion, we further introduce patch-based image corruption to simulate the condition when the nullified data is injected into the client data. We term this injection of nullified images as patch-based corruption. We characterize these corruptions in the following three forms (but note that many others could exist): black patches, patches with Gaussian noise, and generative patches (we use StyleGANv2) <cit.>, illustrated in Figure <ref>. Here, specific data samples in the client's local dataset are substituted with such patches, while the corresponding labels remain unchanged, e.g., a random generative patch has an object label assigned to it.





 §.§ Single-Interaction Discovery of Noisy Clients

Unlike centralized learning,  where the server has direct access to inspect all the data, FL presents a unique challenge due to the decentralized and private nature of the data. In FL, clients contribute to the server by sharing their model parameters rather than raw data <cit.>, making it impossible to visually verify the quality of data. To address this issue, we propose a novel approach that can discover the noisy client with only a single-interaction (or in a one-shot manner) using only a local model. Specifically, we identify the noisy clients by evaluating the gradient norms <cit.> calculated from local data during the initial training round without compromising the client's data confidentiality.


The challenge in training deep neural networks on input (such as images) with noise and distortions is that the model can easily affected by the perturbations, thus producing overconfident predictions on distorted input data samples <cit.> or leading to poor generalization. Motivated by the findings of <cit.>, we empirically discover that a client's local model trained on distorted inputs exhibits distinct behavior in its gradient space compared to a model trained on clean inputs (see Figure <ref>).

In other words, the magnitude of gradients, when trained on clean or corrupted data, is indicative of variation in the model's learning process. The gradient norm effectively encapsulates the relationship between input features and the model’s output. Intuitively, we propose a single-interaction method to estimate data distortion by evaluating the gradient norm of the softmax cross-entropy loss ℒ during the backpropagation phase of local model training that we get at no extra cost. Given the model parameterized by θ, we define the loss of the input batch data 𝐱 with its ground true labels 𝐲 as:

    ℒ(θ; 𝐱, 𝐲) = -log( e^f_𝐲(θ; 𝐱) / T/∑_c=1^C e^f_c(θ; 𝐱) / T)

where the f(·) signifies the model's output function, f_y(θ; 𝐱) is the output logit from the model f for the ground true label 𝐲 given the input 𝐱, and 
f_c(θ; 𝐱) is the output logit from the neural network for class c. The gradient norm of the input batch data 𝐱 over C classes is defined as follows:

    g(𝐱) = ∇_θℒ(θ; 𝐱, 𝐲) _p




The term g(𝐱) refers to the gradient norm of the input batch data 𝐱. Considering the k-th client within a set of clients, the overall local dataset of client D_k is split into B mini-batches. Hence, we compute the gradient norms for the entire local dataset D_k, expressed as { g(𝐱_b) }_b=1^B. After the initial round of local training, the server aggregates the gradient norm results from each of the N clients, forming a score vector by computing variance. By applying the kernel density estimation on the aggregated gradient norm vectors from the clients, we observe a distinct separation in the distributions of clean and noisy clients in Figure <ref>. Subsequently, we apply the K-means clustering on the set of all client gradient norm vectors to distinguish two clusters, as illustrated in Figure <ref>. We then compute the centroid of each cluster, categorizing the one with the higher centroid value as `clean' and its counterpart as `noisy' inspired by the findings of <cit.>. We examine the outcomes of client clustering within the  framework. During the initial training round, the server categorizes all clients into either clean or noisy clusters based on gradient norms. We then visualize 9 images sampled from both clean and noisy clusters. As shown in Figure <ref>, the images from the noisy cluster are contaminated by a mixture of noise, whereas those from the clean cluster remain intact.

A key advantage of our proposed method is its implementation at the beginning of the federated training process, eliminating the need for additional communication rounds. By utilizing only the aggregated gradient norms from each client, our approach is directly usable (as a plug-in) with various FL strategies. Our method incorporates only the gradient norm (i.e., a scalar value) along with standard model parameters during the aggregation phase, thereby not increasing communication overhead. Moreover, this method enhances security by requiring clients to share only scalar values, significantly reducing the risk of sensitive data leakage.







 §.§ Federated Noise Sifting

In FL, the formulation of the global model normally involves the aggregation of local models from each client, weighted based on the total count of data samples in their respective local datasets <cit.>. When applying an average aggregation strategy when clients' data follows the IID setting, the contribution of each client's model parameters to the server is equivalent. Consequently, in scenarios of noisy federated learning, the contribution of noisy clients can seriously degrade the generalization quality of the server-side aggregated model, as detailed in Table <ref>. 

Normally, the optimization function of a standard noisy FL problem is to minimize the loss from both noisy and clean clients defined in Equation <ref>, where the model parameters from each client's model are weighted by the amount of data it has divided by the total number of data samples in all clients. To mitigate the detrimental effect of noisy clients, we propose a noise-sensitive weights aggregation strategy that is resilient to data corruption during the federated training process, termed . Specifically, we introduce the weighting factor α and β to control the contribution of model updates from clean and noisy clients, respectively. Given a set of client updates and their corresponding weights, the global parameter θ is updated by aggregating the weighted updates from all clients, including both clean and noisy ones. Considering θ_g^t as the global parameter at the training round t, K total clients including N clean clients and M noisy clients. Let w_k be the weight and θ_k be the model parameter of client k. Thus, We formulate the proposed global parameter update as follows:

    θ_g^t ←θ_g^t-1 - η(    ∑_n ∈ [K] 
    s.t.  D_n  is cleanw_nα∇ℓ(θ_n^t; D_n) 
       + ∑_m ∈ [K] 
    s.t.  D_m  is noisyw_mβ∇ℓ(θ_m^t; D_m) )

where ∇ℓ(θ_n^t; D_n) represents the gradient of the loss function for the clean client n at training round t, while ∇ℓ(θ_m^t; D_m) denotes the corresponding gradient for the noisy client m in the same training round. η denotes the learning rate. The coefficients α and β are utilized to proportionally scale the model update from clean and noisy clients, respectively. Concisely, our approach involves assigning a higher weighting factor to clean clients and a reduced one to noisy clients to lower their contributions to the overall global model. We provide the complete   algorithm in Algorithm <ref>. 

[ht]

: Our method can be easily plugged into various FL strategies like the standard Federated Averaging. N is local clients, T is the total training rounds, E is the epoch of local training, and η is the learning rate. At each round t, the server will sample S_t clients that can be divided into n clean clients and m noisy clients. (FedAvg, )






§ EXPERIMENTS








 §.§ Datasets, Models, and Configuration


Datasets. To demonstrate the efficacy of , we develop the noisy datasets benchmark based on <cit.>. we consider the image classification task and test our approach across various datasets, including CIFAR10/100 <cit.>, Path-MNIST <cit.>, Fashion-MNIST <cit.>, EuroSAT <cit.>, and Tiny-ImageNet <cit.>. These datasets are further applied data corruptions that are defined in Section <ref> to create the noisy input for FL. Specifically, we categorize data corruption into distortion and patch-based noise. In practice, distortion corruption consists of noise addition (e.g., Gaussian noise, shot noise, impulse noise, pixelated noise), blur (e.g., defocus blur, glass blur, motion blur, Gaussian blur), photometric distortions (e.g., brightness, contrast, saturate), synthetic distortions (e.g., snow, fog, elastic, spatter), and compression artifact (e.g., JPEG compression). Moreover, we further evaluate our method on patch-based obfuscation noise, including black patch, Gaussian noise patch, and StyleGAN-based patch <cit.>.

Baselines and Implementation. We apply  in conjunction with four widely used FL strategies to evaluate its effectiveness, namely FedAvg <cit.>, FedProx <cit.>, FedTrimmedAvg <cit.>, and FedNova <cit.>. We employ ResNet-18 as the main model architecture, utilizing the mini-batch SGD as the universal local optimizer for all FL strategies. Moreover, we further evaluate  on a different neural architecture ConvMixer-256/8 in Table <ref>. The optimizer is characterized by a learning rate of 0.01, an SGD momentum of 0.9, and the weight decay is set to 10^-4. We set the number of local training epochs to 5 and the global communication rounds to 150 across all datasets. 

We consider a setup with N = 20 clients for our experiments unless mentioned otherwise. This choice aligns with standard practices in FL and accommodates our computational limitations. The training set is distributed under both IID and non-IID settings. For the main experiments, we divide the client set into 15 noisy clients and 5 clean clients, with full client participation r_p = 1.0. We compute L_1-norm of the gradient of last layer for noisy client detection in all the cases, see Table <ref> for comparison between the effectiveness of L_1 and L_2 norms, as well as the impact of batch size on the detection performance. We conducted all experiments on NVIDIA A10 GPUs.









 §.§ Results and Ablation Studies


Participation of noisy clients deteriorates the performance of the global model. To validate the efficacy of our proposed method, we first conduct an experiment for model training with clean and noisy input across all the datasets and utilize the same noise configuration for our further empirical evaluation. With this, we aim to evaluate the upper-bound performance that can be achieved when learning from a mixture of noisy and clean clients. Table <ref> presents the comparative results of average accuracy for all considered datasets. We focus on three specific distortions (i.e., defocus blur, Gaussian blur, contrast) due to their significant impact on degrading the model's generalization capability to simulate the worst case in noisy FL.

For the generation of distorted data used in the experiments summarized in Table <ref>, each noisy sample was produced by randomly selecting a distortion type with the configuration characterized by the noise severity level ξ = high. We set the noise level to NL_m = 100% for every client m for the experiments of both Table <ref> and Figure <ref>. We see the participation of noisy clients leads to a significant degradation in the model's generalization capability across all tasks, indicating the detrimental impact of noisy data in the FL environment. Furthermore, due to the inherent lack of visibility into the data from these federated clients, the resultant global model tends to be of low quality and it may become challenging in a real-world setting to identify the underlying reasons for its poor performance.

FedNS significantly improves standard federated aggregation methods. We investigate the robustness of our proposed method by applying  on six image datasets with different settings under the noisy scenario. As shown in Table <ref>, the performance of all aggregation methods exhibits a general trend of improvement by simply plugging FedNS to the considered strategies. In particular, we consider the worst-case with heterogeneous data setting in Table <ref>, where 15 out of 20 noisy clients participate in the federated training with high noise severity and 100% noise level. Adding  to FL strategies yields better overall performance among all the datasets, especially for some vulnerable datasets (e.g., Path-MNIST) that are sensitive to data corruption. Additionally, we demonstrate the efficacy of  in dealing with patch-based noise on CIFAR-10 and Path-MNIST as presented in Figure <ref>. From Figure <ref>, we observe that FedNS consistently boosts the performance of the aggregation method across all the datasets and settings. This further shows that FedNS is capable of handling various types of noises a model can encounter in a real-world setting. 



Impact of noisy client distributions on performance across varying numbers of clients. We evaluate FedNS under the influence of noise client percentages with different amounts of clients at scale, where K denotes the number of clients. We select K from { 30, 50, 100 } and set the noisy client percentage as 30%, 50%, 80% , respectively. As shown in the figure, the results exhibit a consistent pattern of variation in model performance across different scenarios. Specifically, as the percentage of noisy clients increases, there is a gradual decline in test accuracy for both IID and non-IID distributions. However, FedNova+NS (IID) consistently outperforms the other methods, maintaining higher accuracy levels across all settings. This demonstrates the robustness and effectiveness of the FedNova+NS method in mitigating the adverse effects of noisy clients, especially as the number of clients increases.



Impact of Client Participation Rate (r_p). Next, we examine how the client participation rate (r_p) affects the performance of  . We conduct experiments on CIFAR-10 and Path-MNIST, maintaining a fixed noise configuration across all settings. We vary r_p to simulate different device participation patterns, choosing values from r_p ∈0.2, 0.5, 1.0, where r_p = 1.0 denotes full client participation. In practice, the server ensures full client participation during the first training round to identify noisy clients. Figure <ref> provides the results. We observe that incorporating   significantly enhances performance across all client participation patterns. Our experiments suggest that  remains robust to variations in client availability and data quality.



Robustness of  on mixed noise conditions. Next, we investigate the robustness of  under complex noise conditions that involve a combination of different noise types, including the distortions and patch-based noises as described in Section <ref>. Table <ref> shows the performance gains achieved by  are particularly significant in high-variance datasets such as Path-MNIST and Tiny-ImageNet, where FedNova+NS demonstrates substantial improvements over standard aggregation. Notably,  consistently enhances the global model's performance across all evaluated datasets, showcasing its robustness in handling intricate noise scenarios. Our results highlight the effectiveness of  in mitigating the impact of complex noise conditions, where decentralized data may be subject to various types of distortions at the same time.



Evaluation on Weight Factor(β) of Noisy Client. We further investigate the effect of weight factor β on overall performance in FedNS. The weight factor β controls the weights of aggregated noisy client local models, as described in Equation <ref>. We considered various values of β∈{0, 0.1, 0.3, 0.5, 0.7, 1.0 }, where β=0 signifies the exclusion of noisy model weights, and β=1.0 indicates the direct aggregation of model weights. As shown in Table <ref>, the setting of β=0.3 yields the best performance, suggesting an optimal trade-off. Interestingly, the exclusion of noisy model weights (β=0.0) leads to a degradation in the generalization capability of the global model. This observation suggests that incorporating mitigated noisy data enhances the robustness of the global model.











FedNS on Real-world Human Annotation Errors. In this experiment, we extend our investigation to assess the efficacy of FedNS in addressing real-world data quality issues, specifically human annotation errors. While this work focuses on mitigating data corruption in the input space, we identify that FedNS can effectively handle label noise - a well-studied problem in noisy FL. Label noise occurs when data labels are incorrectly assigned while the input features remain unaltered. Specifically, we assessed FedNS on CIFAR-10/100N, two benchmark datasets featuring real-world noisy labels resulting from human annotation errors <cit.>. We employed FedAvg and FedNova as baseline approaches, adhering to the training configuration outlined in Section <ref>. From Table <ref>, we observe that FedNS consistently improved the performance across all the experiments. Our findings demonstrate that FedNS not only excels in mitigating input space corruption but also shows promising results in handling label noise. The ability to address both input space corruption and label noise positions FedNS as a valuable tool for practitioners dealing with real-world datasets, where multiple types of data imperfections may coexist.



Impact of Initial-Round Client Participation on FedNS Performance. In FL, the assumption that all clients must be warmed up and participate from the very first training round may not always hold true, as some clients may only participate in later rounds. To address this, we explore the scenario where only a subset of clients participates in the initial round. Instead of collecting gradient norms solely in the first round, we iteratively apply FedNS until all clients have been engaged. We evaluate this approach by setting the first-round participation rates to 10%, 50%, and 100%, with 100% representing full client participation. The results, presented in Table <ref>, highlight the robustness of FedNS across varying levels of initial-round client participation.



FedNS on Different Noise Level (NL) of Client Data. In Table <ref>, we examine the performance of  across various noise configurations. Specifically, we consider noise levels NL∈{ 50%, 80%, 100%} and noise severity ξ∈{ Medium, High }. These results demonstrate that  substantially enhances model generalization under high noise conditions, especially when clients' data is completely corrupted (i.e., 100% noise level and high severity). Conversely, in scenarios with milder noise where the impact on federated models is minimal, the implementation of  does not negatively affect the generalization process. Hence, we limit our experiments to noise levels at or above 50%, as noise levels below this threshold have negligible impact on model performance.



Hyperparameters Selections for Noisy Clients Detection. One of the essential components of  is using gradient norms to identify noisy clients. We investigate several factors that may affect the performance of detecting noisy clients, specifically the selection of the L_p-norm and the batch size of gradient norms for clustering. This ablation experiment is performed on the CIFAR-10 and Path-MNIST datasets, as presented in Table <ref>.

The comparison between the L_1-norm and L_2-norm shows that the L_1-norm generally outperforms higher-order norms. Moreover, we evaluate the impact of batch size by setting the mini-batch size to range from 1 to 128. Our experiments suggest that a larger batch size effectively reduces the variance of the gradient estimates. The results, shown in Table <ref>, indicate that a batch size selection around 16 ∼ 64 yields better performance across all settings.



Evaluating  with alternative model architecture. In this experiment, we investigate the robustness of  when applied to a different model architecture. We employ the ConvMixer-256/8 <cit.> model and train it using FedNova on a range of datasets. The noise configuration remains consistent with the details provided in Section <ref>, and we evaluate the model's performance under both IID and non-IID settings. As shown in Table <ref>, the federated aggregation method, when paired with , achieves enhanced performance across all considered datasets. These improvements, observed consistently across tasks, demonstrate the adaptability and effectiveness of  when integrated with alternative architectural paradigms. The results highlight the versatility of our approach in enhancing the performance of different federated model regardless of the neural architecture, further emphasizing its potential as a robust and flexible method for FL.





§ CONCLUSION




In this paper, we study the training of the federated neural networks within a noisy environment in which the input space of client data is corrupted. We propose to utilize the gradient norm of local training updates to discover the noisy clients. We then propose , an effective method designed to integrate with diverse FL aggregation methods to mitigate the impact of model updates from noisy clients. Our experimental results on several benchmark datasets show that  is robust against data perturbation, which can significantly boost the generalization of federated models. We further present extensive ablation studies to provide a better understanding of . Our findings highlight the importance for FL approaches that prioritize robustness when training models on decentralized data. By focusing on this aspect, we can enhance the reliability and stability of FL in a real-world setting, where data quality may vary significantly across participating clients. Future work will investigate 's efficacy in the language modeling domain and learning from multi-modal noisy data in a decentralized setting.



IEEEtran


