CoAst: Validation-Free Contribution Assessment for Federated Learning based on Cross-Round Valuation
Hao Wu Likun Zhang Shucheng Li Fengyuan Xu Sheng Zhong




0000-0002-0980-9805

National Key Lab for Novel Software Technology, Nanjing University
Nanjing
Jiangsu
China
hao.wu@nju.edu.cn


0000-0001-9309-4541

University of Chinese Academy of Sciences
Beijing
China
zhanglikun@iie.ac.cn


0000-0002-5414-6203

National Key Lab for Novel Software Technology, Nanjing University
Nanjing
Jiangsu
China
shuchengli@smail.nju.edu.cn


Corresponding author.
0000-0003-3388-7544

National Key Lab for Novel Software Technology, Nanjing University
Nanjing
Jiangsu
China
fengyuan.xu@nju.edu.cn


0000-0002-6581-8730

National Key Lab for Novel Software Technology, Nanjing University
Nanjing
Jiangsu
China
zhongsheng@nju.edu.cn






§ ABSTRACT 
In the federated learning (FL) process, since the data held by each participant is different, 
it is necessary to figure out which participant has a higher contribution to the model performance.
Effective contribution assessment can help motivate data owners to participate in the FL training.
Research works in this field can be divided into two directions based on whether a validation dataset is required.
Validation-based methods need to use representative validation data to measure the model accuracy, which is difficult to obtain in practical FL scenarios.
Existing validation-free methods assess the contribution based on the parameters and gradients of local models and the global model in a single training round, which is easily compromised by the stochasticity of model training. 
In this work, we propose CoAst, a practical method to assess the FL participants' contribution without access to any validation data.
The core idea of CoAst involves two aspects: one is to only count the most important part of model parameters through a weights quantization, and the other is a cross-round valuation based on the similarity between the current local parameters and the global parameter updates in several subsequent communication rounds.
Extensive experiments show that CoAst has comparable assessment reliability to existing validation-based methods and outperforms existing validation-free methods.
   


<ccs2012>
   <concept>
       <concept_id>10002951.10003227.10003251</concept_id>
       <concept_desc>Information systems Multimedia information systems</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
   <concept>
       <concept_id>10010147.10010178.10010224.10010225</concept_id>
       <concept_desc>Computing methodologies Computer vision tasks</concept_desc>
       <concept_significance>500</concept_significance>
       </concept>
 </ccs2012>


[500]Information systems Multimedia information systems
[500]Computing methodologies Computer vision tasks












  
: Validation-Free Contribution Assessment for Federated Learning based on Cross-Round Valuation
    Sheng Zhong
    September 9, 2024
===============================================================================================



§ INTRODUCTION


With the development of deep learning (DL), the concept that "Data is the new oil" has gained more and more consensus among people <cit.>. 
The emerging remarkable capabilities demonstrated by large language models <cit.> further bring attention to the enormous value of collaborating on a large amount of data.  
To train DL models on data owned by different parties, collaborative learning techniques, represented by federated learning (FL) <cit.>, have been extensively studied.
However, due to disparities in the data held by different participants, including variations in data quality and quantity, each participant's contribution to the performance of the FL model differs a lot. 
How to accurately evaluate the contribution of each participant is crucial for the fair distribution of rewards. This process benefits the promotion of data quality and creates incentives for data sharing  <cit.>.


 




There has been a line of research in the community on the contribution assessment of participants in FL, which can be mainly divided into two categories, i.e., validation-based methods <cit.> and validation-free methods <cit.>.
The effectiveness of validation-based methods heavily relies on a representative validation dataset, which is used to evaluate the model performance.
However, in real-world FL scenarios, obtaining the representative validation dataset that covers the distribution of all clients' data can be challenging.
To overcome the limitations imposed by the validation dataset, validation-free methods are proposed to assess the contribution based on the statistical characteristics of model parameters. 
They estimate the parameters' correlation, information gain, and mutual information among local models (produced by clients) and the global model (aggregated by the server) to answer whose contribution is higher.


However, the existing validation-free efforts, detailed in the next section, only consider the models' parameters (or gradient) in a single training round. 
We refer to a training round as the process of the client updating and uploading the local model and getting the global model from the server after aggregation. 
Due to the difficulties, such as the stochastic nature of gradient descent and the uninterpretable nature of the DL model, 
the accuracy of these assessment methods can sometimes be seriously compromised.
Performing validation-free assessment faces two challenges:
1) The training process of DL models does not update parameters in a linear way, so only comparing the parameters of the local model in each round with the parameters of the final model will ignore the clients' contribution reflected in the iterative process of parameter updating.
2) Due to the parameter redundancy of DL models, not all parameters in the local model reflect the client's contribution to the performance improvement of the global model. Unimportant local parameters may comprise the assessment's effectiveness.

In this work, we propose , a validation-free FL contribution assessment method, performing client valuation in a cross-round way.
The overview of is depicted in Figure <ref>.
The core idea of the cross-round valuation is to evaluate a local model's contribution in a certain round (e.g., t) leveraging the parameter updates of the global model in several subsequent rounds (e.g., rounds {t+1, …, t+k}).
To cope with the interference of unimportant parameters, shares a similar idea with the ternary weights quantization <cit.>.
spotlights the most important parameters by neglecting the parameters with small updates and keeping only the sign of the parameters with a large update.
It is worth noting that our assessment goal is to accurately and effectively measure each client's contribution to the FL model, rather than improve the FL model performance.
As a result, neither affects the local training procedure nor changes the global models.
Experimental results show that the assessment accuracy of is comparable to the validation-based methods and outperforms the existing validation-free methods.
We summarize the contribution as follows. 

	
  * We design a novel validation-free contribution assessment method, , which can answer whose contribution is greater in a practical FL scenario without a validation dataset.
	
  * We propose a cross-round assessment mechanism to consider the effect of the intermediate local models on the final trained model, and we utilize the ternary weight quantization technique to capture the parameters that contribute the most.
	
  * Experimental results demonstrate that outperforms the SOTA validation-free methods in assessment effectiveness and achieves comparable performance to validation-based methods.




The rest of the paper is organized as follows. 
In Section <ref>, we review the related works.
In Section <ref>, we formalize the targeted scenario and problem. 
Section <ref> presents the detailed design of our proposed . 
Then, we describe the experimental settings in Section <ref> and provide the evaluation results in Section <ref>. 
The limitation of this work is discussed in Section <ref>.
Finally, we conclude this paper in Section <ref>.





















 


§ RELATED WORKS



Validation-based methods.
The validation-based methods use a validation dataset to assess a client's contribution by evaluating its impact on the performance of the aggregated model.
The leave-one-out is the most natural way to assess the value. 
It assesses the data value of one contributor by calculating the model performance change when the contributor is removed from the set of contributors.
However, the leave-one-out is unfair to multiple similar and mutually substitutable contributors.
Ruoxi et al. <cit.> use the Shapley value to assess data value in the FL scenario.
They compute the marginal increase of the average accuracy of the model due to the addition of one data contributor.
Guan et al. <cit.> extend the application scenarios of  Shapley value-based solutions to FL scenarios. 
Zhenan et al. <cit.> apply the Shapley value-based solutions to vertical federated learning and improve the efficiency through approximation.
Zhaoxuan et al. <cit.> allow for efficient data valuation without long-term model training. 
They assess the contribution through a domain-aware generalization bound, which is derived from the neural tangent kernel (NTK) theory.
There are also research efforts to improve the system performance <cit.> and to analyze the fairness <cit.>.
When the server uses this line of work to assess the clients' contribution, it requires a representative dataset which covers the distribution among all clients' data. 
However, in real-world FL scenarios, obtaining such a representative validation dataset is infeasible.


Validation-free methods.
The validation-free methods use statistics of training data or the correlation among local and global parameters to value the clients. 
These works usually have some specific assumptions on the distribution of gradients, model parameters, or local training data.
Therefore, they may face performance degradation in the real world when their assumptions are not satisfied.
Xinyi et al. <cit.> propose a volume measure on the replication robustness, which assesses the contribution based on the diversity of training data.
However, work <cit.> shows that this idea not only suffers from exploding volumes in high-dimensional inputs but also entirely ignores the useful information in the validation dataset.
Rachael et al. <cit.> measure the data value based on the information gains of the model parameters.
They hold that the contributors with the highest value can reduce the uncertainty of the model parameters.
However, when the distribution of data is complex, the accuracy of the information gains is biased.
Xinyi et al. <cit.> use the gradient similarity to measure the data value of the contributors' combination by comparing the data of one combination of the contributors with the gradient similarity of the global FL model trained by all contributors.
However, due to the randomness of the stochastic gradient descent and gradient pruning, the value assessed in some rounds may not accurately reflect the true value of the data, or even the value of high-value data is negative. 
Hongtao et al. <cit.> propose a test data-free data value evaluation based on the pairwise correlation among the models based on the statistical characteristics of the models.
They assume that the parameters trained by different contributors share the same distribution, which may not be satisfied when the data is imbalanced and non-independent, and identically distributed.



§ PROBLEM OVERVIEW






 §.§ Targeted Scenario


We assume all participants, including clients and the server, are honest and follow the agreed-on training protocol of FL. 
Due to differences in training data quality and quantity among clients, each client's contribution to overall model performance varies.
The server can access the model parameters uploaded by each client, but it lacks a representative validation dataset.
Each client delegates the server to evaluate the contribution of each client without offering validation data.
Without loss of generality, we assume that each client is involved in all training rounds.
The training data of each client is prepared before the FL training, and they will not add new data during the training process.




 §.§ Problem Formalization


Consider an FL training procedure with one server and N clients. 
The training procedure consists of M training rounds.
Recall that, in one training round, a client uploads its local parameters to the server once and receives the corresponding aggregated parameters (a.k.a. global parameters).
After M training rounds, the contribution of all clients is determined. 
We denote the ground-truth contribution of all clients as P={p_1, p_2, …, p_N}, where p_i is the contribution of client i. 
The ranking of all clients' contribution is denoted as R = {r_1, r_2, …, r_N}, and

    r_i = |{ j | p_j ≥ p_i  }|,

where |·| returns the number of elements in a set.


Our goal is to design a function L, which can measure how much each client improves the performance of the global model.
That is, 

    P̂ = L(Θ,  {θ_i^t}_ i∈ [1,N],t∈ [1,M]),

where Θ denotes the parameters of the global model ϕ_Θ in the last round, and θ_i^t denotes the parameters of the local model trained by client i in round t.
Then the predicted R̂ can be calculated based on the P̂ through Equation <ref>.
The objective of function L is to minimize the distance of the predicted R̂ and R, i.e., 

    min d(R̂, R),

where d(·, ·) is the distance measurement function.

Due to the multi-round property of FL algorithm, the contribution score of each client in round t (denoted as P̂^t) can be naturally represented as:

    P̂_i^t = L(Θ^t,  {θ_i^t}),

where Θ^t is the parameters of the global model aggregated in round t.
So, we can reformulate Equation <ref> as:

    P̂  = {∑_t=0^M P̂^t_i | i∈ [1,N]}.













§ DESIGN







 §.§ Design Overview


We propose two key techniques to address the challenges introduced in the introduction section. 
First, we design a cross-round valuation mechanism.
Due to stochastic gradient descent,  gradient pruning, and parameter regularization, the parameter updates in a certain round (e.g., round t) may not accurately reflect the true value of the client, and even high-value clients may be assigned negative contributions.
Fortunately, the FL training process minimizes the optimization objective and improves the model's accuracy, which means that the global parameter updates have a positive contribution in most training rounds.
It allows us to value the client i in round t with global parameters of subsequent several rounds.

Second, we borrow ideas from efforts on model compression and quantization to filter out those unimportant parameters. 
Binary weight quantization <cit.> is proposed for the efficiency of computation and storage and demonstrates that retaining only the sign of parameters during model updates can still reach acceptable accuracy.
Then, the ternary weight quantization <cit.> sets unimportant parameter updates to zero on top of binary weight quantization and reaches comparable accuracy to full precision training. 
It shows that by setting a threshold, parameter updates that contribute minimally to the model performance can be effectively filtered out.
Thus, we apply the idea of ternary weight quantization to the global model aggregation and then value the client's contribution with the remaining important parameters. 


 


We demonstrate the workflow of in Figure <ref>. 
The contribution assessment is transparent to clients, and there is nothing to change during the local training procedure.
On the server, when receiving the local parameters trained in round t, first prunes the parameters by their importance through the ternary weight quantization, then performs the aggregation on the pruned models.
After aggregation, we use cross-round valuation to measure the contributions of local models.
In the next part, we will detail these two key designs. 










 §.§ Parameter Pruning



Without loss of generality, we use the training process of round t as an example to detail the algorithm design.
N clients first locally train the local models of round t based on the global parameters of round t-1, denoted as Θ^t-1. 
Then, all clients upload local parameters of round t to the server, which are denoted as {θ_i^t}_i∈{1,…,N}.
In the following procedure, clients do nothing but wait for the aggregated global parameters of round t, denoted as Θ^t, to continue their local training in the next round.

Once all local parameters {θ_i^t}_i∈{1,…,N} are received, calculates the local updates of round t, denoted as {Δ_i^t }_i∈{1,…,N}, where  

    Δ_i^t = θ_i^t - Θ^t-1 .

Then performs parameter pruning according to their importance, similar to the idea of model quantization such as binarization and ternarization.
Considering the structural and functional differences between layers of DL models, quantifies the parameter importance in a layer-wise manner.  
Here we denote the parameter updates in each layer as Δ_i := [δ_1, δ_2, …, δ_l] where l refers to the number of layers.
Take the j-th layer as an example. 
first calculates the r-th percentile of |δ_j|, denoted as δ_j^r, where r is a hyperparameter controlling the pruning rate.
Then, prunes the parameters by clipping the parameter updates. 
For each element of δ_j, denoted as u, it is clipped as:

    ũ =
    	
    		1      if  u > δ_j^r 
    
    		-1      if  u < - δ_j^r 
    
    		0      otherwise.

It means we regard the elements of δ_j whose absolute values are greater than δ_j^r as important and only keep their signs, while the remaining elements are pruned to 0.
Given the clipped parameter updates Δ̃_i^t, the pruned local parameters θ̃_i^t can be calculated as

    θ̃^t_i = Θ^t-1 + α·Δ̃_i^t,

where α is a hyperparameter for normalization.
We report the whole parameter pruning procedure in Algorithm <ref>.







 §.§ Cross-round Valuation



 
In each round, the contribution of the model is valued by the global model aggregated of the next k rounds, and it also values the local models of the last k rounds.
Recall that the parameters of the global model are the average of the pruned local parameters (Equation <ref>), which are calculated by adding the sign of the parameter update. 
Therefore, the global parameters Θ^t is calculated by

    Θ^t = 	1/N·∑_i=1^N θ̃_i^t = Θ^t-1 + α/N·∑_i=1^N Δ̃_i^t .

Therefore, in round t, the parameter update of the global model denoted as U^t := Θ^t - Θ^t-1, is proportional to the sum of the pruned local updates, i.e., 

    U^t  ∝∑_i=1^N Δ̃_i^t .

Recall that the value of element ũ∈Δ_i' belongs to {-1, 0, 1}. 
That is, the U^t (Equation <ref>) is the normalized voting result, which indicates, for each element b ∈Θ^t-1, how many clients believe that its value should be increased by α/N, and how many clients believe that it should be decreased by α/N.

After k rounds following the t-th round, we obtain the parameters Θ^t+k , which can be calculated by 

    Θ^t+k = Θ^t + ∑_e=t+1^t+k U^e .

In the following part, we denote ∑_e=t+1^t+k U^e as U^(t,k) for simplification.
Since the global parameters are obtained by averaging all local parameters, we can regard the parameters of any client as a correction to the sum of parameters of the other N-1 clients.
Without loss of generality, for any client i, we reformulate Equation <ref> to 

    Θ^t+k = 1/N·∑_j∈{1,…,N}∖{i}θ_j^t +  1/N·θ_i^t  + U^(t,k) .


We assume that the model's performance is improved after the k rounds.
That is, among all clients, if a client's local model of round t, i.e., θ_i^t, during the parameter aggregation process is more similar to the subsequent global parameter updates, i.e., U^(t,k), then the contribution of this client in this training round  is greater.
We demonstrate this idea in Figure <ref>.
Clients will recalculate the local models based on the last global model, so the choice of k should not be too large. 
Otherwise, the update direction represented by U^(t,k) may not capture the details of stochastic gradient updates and thus cannot indicate the contribution in one training round. 

 



Therefore, the contribution of client i can be measured by the similarity between θ_i^t and U^(t,k). 
Here, we use Signed Cosine similarity to measure the similarity because the θ_i^t and U^(t,k) are local parameters and global updates, respectively. 
Note that Signed Cosine similarity is sensitive to the sign information of vectors and can better reflect the directional relationship between vectors. 
Due to the parameter pruning, the model updates can be considered as the sign of each parameter's update (Equation <ref>).
That is, in our design, the sign of these updates is more important than their magnitude. 

Note that θ and U^(t,k) share the same shape, and we assume that they can be indexed through h. 
calculates the client's contribution in round t by 

    p̂_i^t = ∑_h=1^|Θ|𝗌𝗀𝗇(θ_i^t[h]) ·𝗌𝗀𝗇(U^(t,k)[h]) ,

where 𝗌𝗀𝗇(·) is the function to indicate the sign of the value.












§ IMPLEMENTATION







 §.§ Dataset Settings


We evaluate the 's performance on three datasets, i.e., CIFAR-10 <cit.>, CIFAR-100 <cit.>, and STL-10 <cit.>.
We randomly partition the training dataset among each participant.
We assume that by randomly and evenly partitioning these three datasets according to the number of clients, several datasets with the same data valuation can be obtained.
Therefore, if a group of clients uses these partitioned datasets for training, the contribution of these clients is the same.
We have set up four scenarios to mimic the contribution differences caused by data quality and quantity differences.
N in the following settings denotes the number of clients.



  §.§.§ Setting 1: Different quantity

Assuming that randomly partitioned datasets share a similar distribution, the more samples in the training dataset, the higher the contribution to the model accuracy.  
In this scenario, we prepare datasets with different contributions by randomly assigning different numbers of samples to each client.
Let the number of clients be N and the size of the training dataset be |X_train|. 
Then, the size of the dataset for the i-th client is 
D_i = 1 - 0.5 ·i/N|X_train|.






  §.§.§ Setting 2: Adding noise

In this scenario, we prepare the training datasets of different qualities for clients by adding Gaussian noise of different intensities.
We first randomly and evenly partition the dataset according to the number of clients.
Then we perform the Gaussian noise to the dataset of client i with a mean of μ_i and standard deviation of σ_i, where i∈{1, …, N}.
We set the mean and variance of Gaussian noise decrease linearly, i.e.,
μ_i = 0.01 · i, σ_i = 0.625 ·i/N.
We report some data samples in Figure <ref>.




  §.§.§ Setting 3: Adjusting resolution

In this scenario, we mimic the data of different quality by adjusting the resolution of the training data through Gaussian blur.
Different degrees of Gaussian blur can be achieved by setting kernels of different sizes and variances. 
We first randomly and evenly partition the dataset according to the number of clients.
Then, we use different degrees of Gaussian blur to preprocess the training data. 
Let the sequence of kernel sizes and standard deviation be s_i and σ_i, where i∈{1, …, N}.
We select a linearly decreasing sequence of kernel sizes and standard deviation, i.e.,
   s_i = 2· i + 1, σ_i = 0.4 · i + 1.
We report some data samples in Figure <ref>.



  §.§.§ Setting 4: Masking

In this scenario, we prepare the dataset with different quality by adding a mask on training data. 
The content covered by the mask is set to 0.
We first randomly and evenly partition the dataset according to the number of clients.
Then, for each client, we randomly mask a part of the image. 
The area of the mask covers r_i% of the image for client i, and its position is randomly generated. 
The r_i is a random number between l_i and u_i, where 
    l_i = 0.5 ·i/N, u_i = 0.75 ·i/N .
We report some data samples in Figure <ref>.




 §.§ Implementation Details


We perform experiments with Pytorch on a server with two A100 (80G) GPU cards.
We use three model architectures, i.e., TinyResNet, ConvNet <cit.>, and ResNet-4.
The TinyResNet consists of a convolution layer, whose weight shape is 3×7×7×64, and a ResBlock <cit.>, whose kernel size is 3×3 and output channel is 64.
The ResNet-4 consists of 4 ResBlocks, whose kernel size is 3×3 and output channel is 64, 128, 128, and 128. 
In our setting, we use 1 central server and 5 clients, (i.e., N=5).
We initialize the learning rate to 0.01 and  gradually decrease it as the training progresses.  
We use the FedAvg method to aggregate the local models.  







 §.§ Baseline Methods

We use three baseline methods.
The validation-based method <cit.>, denoted as baseline 1, measures the accuracy of the aggregated models w/ or w/o one local model with the validation dataset to calculate the contribution. 
Although baseline 1 is computationally time-consuming, it has the highest accuracy among Shapley value-based solutions by exhaustively considering all possible cases.
We implement two SOTA validation-free methods, i.e., Fed-PCA <cit.> and CGSV <cit.>, as the baseline 2 and baseline 3.






 §.§ Metrics


To measure the accuracy of the contribution assessment,  we use the Spearman correlation coefficient <cit.> as the distance measurement function d in Equation <ref>. 
The Spearman correlation coefficient is good at measuring the degree to which the ranks of the two variables are associated with each other.
And it also is used in related works <cit.>.
Formally,  R and R̂ are two sequences of length n, which denote the ground-truth order and predicted  order of the clients' contribution, i.e., 

R = [o_1, o_2, … , o_n], R̂ = [ô_1, ô_2, … , ô_n].

We calculate the Spearman correlation coefficient (ρ) through the:

    ρ = 1 - 6/n(n^2-1)∑_i=1^n ‖ o_i - ô_i ‖_2.

The ρ ranges from -1 to 1, where -1 indicates a perfectly negative correlation, i.e., sequences R and R̂ are in reverse order, 0 indicates no correlation, i.e., random guessing, and 1 indicates a perfectly positive correlation.








§ EVALUATION



We measure the performance of our in four configurations with three datasets and three model architectures.
Configuration 1 is CIFAR-10 and TinyResNet; 
Configuration 2 is CIFAR-100 and TinyResNet; 
Configuration 3 is STL-10 and ResNet-4; 
Configuration 4 is CIFAR-100 and ConvNet. 



 §.§ Overall Performance



In the 's experiments, for hyperparameters in Algorithm <ref>, we set r to 10, α to 0.02, and N to 5. 
We set the k in Equation <ref> to 2. 
We report the overall performance in Table <ref>. 
SV, PCA, and CGSV represent baseline 1, baseline 2, and baseline 3, respectively.

The average accuracy of baseline 1 and are 0.855 and 0.9, which means that our is comparable to that of baseline 1, which is a validation-based method.
even outperforms baseline 1 by 0.22, 0.02, and 0.12 in setting 1, setting 2, and setting 4.
Our 's performance outperforms the SOTA validation-free methods, i.e., Fed-PCA (baseline 2) and CGSV (baseline 3),  in almost all cases.
On average, our outperforms Fed-PCA by 0.94 and CGSV by 0.52 in all cases.
The poor performance of Fed-PCA is because the model architecture used in our experiment is too complex to perform precise probability analysis.
Experimental results demonstrate the  's effectiveness and robustness in different cases.




 §.§ Ablation Study






  §.§.§ k in cross-round valuation


We explore how k affects the performance of .
We perform the experiment with different k values, which means that different numbers of global updates are used to assess the local models' contribution.
We report the results in Table <ref>. 
In different experimental settings, with k=5 reaches the best performance, and the performance of with k=2 is comparable to that of with k=5.
The small and large k values, i.e., k=1 and k=10, have relatively poor performance. 
This is because a small k value may still let the contribution assessment be affected by stochasticity, while a large value of k may make lack attention to the stochastic gradient descent process.
We recommend setting k to 2 or 5 when using our in practical use.


    


  §.§.§ Parameter pruning




    


Recall that the parameter pruning consists of a parameter update clipping procedure (Line <ref> in Algorithm <ref>) and a top r% update selection procedure (Line <ref> in Algorithm <ref>). 
To measure the effect of parameter pruning, we perform the following contrast experiments with different settings, which are as follows.

	
  * Ours. r=10, w/ update clipping.
	
  * Exp1. r=20, w/ update clipping.
	
  * Exp2. r=10, w/o update clipping.
	
  * Exp3. r=100, w/o update clipping. (No parameter pruning.)

We report the results of these contrast experiments in Table <ref>.
By comparing the results of ours and Exp1, we can conclude that increasing the proportion of parameter selection leads to a slight decrease in performance, which is likely due to the noise introduced by the selected parameters.
Our 's average performance is 0.40625 and 0.25625 higher than Exp2 and Exp3, respectively, indicating that the design of parameter pruning effectively ensures the accuracy of contribution assessment.




  §.§.§ Update clipping strategy




In the , we use a hyperparameter α (Equation <ref>) to normalize the local parameter update in each round. 
Here we explore how the hyperparameter α value affects the contribution assessment effectiveness. 
We perform three experiments with CIFAR-10 and TinyResNet.
We set M to 100, N to 5, and r to 10.
In the first two experiments, we set the value of α to 0.01 and 0.02, respectively.
In the third experiment,  we use an adaptive clipping strategy, where we set α as the average value of the selected r% parameters (i.e., 10) of each layer.
We report the results of these three experiments in Table <ref>.
By comparing the experimental results of the first two experiments, it can be seen that the choice of hyperparameters has little impact on the assessment performance.
However, in the third experiment, the assessment performance decreases. 
This is because clipping the parameter update to different values interferes with the assessment process. 
In our design, we aim to quantize all parameter updates and assess the contribution based on the direction of the local parameters and global parameter updates.




  §.§.§ Client number

Here we explore how the number of clients, i.e., N, affects the stability of . 
We experiment with CIFAR-100 and TinyResNet and set k to 10 and r to 10.
We report the experimental results in Table <ref>.
When the number of clients increased from 5 to 10, the performance change of Fed-PCA averaged 0.4775, the performance change of CGSV averaged 0.38, and the accuracy change of our method averaged 0.07.
When the number of clients is 5, our outperforms Fed-PCA and CGSV by an average of 1.025 and 0.5225, respectively. When the number of clients is 10, our outperforms Fed-PCA and CGSV by an average of 0.775 and 0.795, respectively.
The experimental results fully demonstrate the stability of our method with respect to changes in the number of clients.















§ DISCUSSION AND LIMITATION




Performance of model pruning.
Recall that in the procedure of parameter aggregation, prunes local parameters according to their importance.
Although weight quantization methods are often used in practical FL scenarios to reduce network bandwidth, they tend to incur performance degradation of the model as well as slower convergence speed.
Thus the convergence time of our is slightly longer than that of the typical method, i.e., without applying any quantization. 


We report the model performance trained through the typical method and ours in Table <ref>. 
The model trained through the typical method converges after 100 rounds (i.e., M=100), while the model trained in our method requires 200 rounds (i.e., M=200) to converge. 
As can be seen, the performance of the model trained through outperforms that of the model trained through the typical method at the 100th round only in one case.
When the model converges, the performance of the model trained through exceeds the model trained in the typical scenario in two cases.
However, experiments in Section <ref> show that weight quantization is important to the accuracy of the assessment since quantization mitigates the negative impact of redundant model parameters on the assessment. 
In future work, we will draw on research ideas in model quantization to improve the model performance in accuracy and convergence.





Application value of contribution assessment.
Concerns regarding data quality assessment and contribution evaluation have become ubiquitous among nations, communities, and individuals alike. 
Some countries have timely developed advanced standards and sophisticated sophisticated indicators to guide norms for contribution and data quality assessment <cit.>. 
The technologies presented in this work could provide pivotal support for the application of these standards within industrial contexts.





§ CONCLUSION



This work proposes a validation-free contribution assessment, , for FL scenarios. 
Compared with existing efforts, it greatly improves the contribution assessment performance under different dataset settings by introducing two key designs: parameter pruning and cross-round valuation.
Comprehensive evaluations showed that our outperforms existing methods on different dataset settings and models.
We believe that will inspire the community to improve the FL paradigm with an inherent contribution assessment.





This work was supported in part by the National Key R&D Program of China under Grants 2022YFF0604503, in part by NSFC under Grants 62272224, 62341201, 62302207, 62272215 and 61872176, in part by the Leading Edge Technology Program of Jiangsu Natural Science Foundation under Grant BK20202001, and in part by the Science Foundation for Youths of Jiangsu Province under Grant BK20220772.



ACM-Reference-Format


