# Fed-GraB: Federated Long-tailed Learning with Self-Adjusting Gradient
Balancer

Zikai Xiao1 Co-first author. Zihan Chen211footnotemark: 1 Songshang Liu1
Hualiang Wang3 Yang Feng4 Jin Hao5 Joey Tianyi Zhou6 Jian Wu1 Howard Hao
Yang122footnotemark: 2 Zuozhu Liu1
1Zhejiang University,
2Singapore University of Technology and Design,
3The Hong Kong University of Science and Technology,
4Angelalign Technology Inc,
5State Key Laboratory of Oral Diseases, Sichuan University,
6Centre for Frontier AI Research (CFAR), A*STAR , Singapore
<EMAIL_ADDRESS>Co-Corresponding author.

###### Abstract

Data privacy and long-tailed distribution are the norms rather than the
exception in many real-world tasks. This paper investigates a federated long-
tailed learning (Fed-LT) task in which each client holds a locally
heterogeneous dataset; if the datasets can be globally aggregated, they
jointly exhibit a long-tailed distribution. Under such a setting, existing
federated optimization and/or centralized long-tailed learning methods hardly
apply due to challenges in (a) characterizing the global long-tailed
distribution under privacy constraints and (b) adjusting the local learning
strategy to cope with the head-tail imbalance. In response, we propose a
method termed Fed-GraB, comprised of a Self-adjusting Gradient Balancer (SGB)
module that re-weights clients’ gradients in a closed-loop manner based on the
feedback of global long-tailed prior derived from a Direct Prior Analyzer
(DPA) module. Using Fed-GraB, clients can effectively alleviate the
distribution drift caused by data heterogeneity during the model training
process and obtain a global model with better performance on the minority
classes while maintaining the performance of the majority classes. Extensive
experiments demonstrate that Fed-GraB achieves state-of-the-art performance on
representative datasets such as CIFAR-10-LT, CIFAR-100-LT, ImageNet-LT, and
iNaturalist. Our codes are available at
https://github.com/ZackZikaiXiao/FedGraB.

##  1 Introduction

Federated learning (FL) is an approach for massively distributed clients to
train a global machine learning model collaboratively without exposing their
private data, which has garnered ever-increasing attention in academia and
industry alike [1, 2]. Unlike conventional machine learning methods, FL brings
the learning objective directly onto the end-user devices for local training,
where only the intermediate parameters (e.g., gradients) need to be sent to a
server for model aggregation and update. This approach not only substantially
reduces the communication overheads but, more importantly, facilitates the
clients in obtaining a generic global model without sharing their private
data, thereby contributing to the development of trustworthy intelligent
systems [3]. Despite its great potential, realizing FL also requires one to
overcome new hurdles in real-world implementations, among which a particular
challenge stems from data heterogeneity, i.e., the non-IID distribution and
imbalanced dataset sizes across clients [4, 5, 6, 7]. These factors often give
rise to a pernicious issue where the data set of different clients jointly
follows a global long-tailed distribution, which is commonly known as the
Federated Long-Tailed learning (Fed-LT) problem [8, 9]. For instance,
patients’ diagnosis varies substantially across medical centers but
collaboratively form long-tailed distributions for certain diseases [10, 11].
In a broad range of applications, performance on the minority classes (i.e.,
tail classes) [12], such as rare diseases, dangerous behaviors in autonomous
driving [13, 14], and abnormal breath or heart rates in wearable devices [15],
play a pivotal role in developing reliable and robust solutions to an FL
system.

Figure 1:  (a): An illustration of the Fed-GraB framework, in which SGB is
mounted on all classes of each client based on the prior vector
$\mathcal{P}_{c}$ derived by the DPA module, ensuring federated balanced local
training; (b): Comparison for the difference of positive and negative
gradients.

The difficulty in addressing the Fed-LT problem primarily stems from the fact
that although clients’ datasets can be locally balanced, globally, i.e., if
the datasets were aggregated, the distribution may be long-tailed. As clients
would not disclose their data information due to privacy concerns, identifying
the global long-tailedness in data distribution could be onerous. To that end,
a natural question arises:

Figure 2: An example of the global and local distributions of CIFAR-10-LT
(Left) and CIFAR-100-LT (Right) under non-IID setting. The bar on the top
demonstrates the sorted global long-tailed distribution. Each row represents
the diverse statistics of class imbalance of each client.

Question 1: How to leverage the global long-tailed statistics, especially for
the minorities that need to be carefully tailored, without conflicting with
privacy concerns?

On the other hand, even if the globally long-tailed data information is
available, existing methods do not directly apply to address such a problem.
Specifically, typical FL methods cope with data heterogeneity via generic
techniques such as dynamic regularizations, client selections, data
augmentation, distillation, and personalization training [16, 17, 18, 19, 4,
20, 21]. These approaches ignored the diverged/imbalanced class levels that
can result in the global long-tailed issue [22, 23]. Hence, they cannot
guarantee decent performance for the tail classes. A key ingredient to improve
long-tailed learning is to boost the performance of tail classes without
degrading the performance of the head classes [24, 25]. While several learning
methods dedicated to long-tailed data exist, they usually assume global class
priors for re-balancing, which are not available in the context of FL due to
privacy concerns [8]. Moreover, these methods may not be applicable in local
training as local distributions could present diverse long-tailed
characteristics and/or are not even long-tailed [9]. As such, it begs another
question as follows:

Question 2: How to perform local training to synergistically aggregate a
global model that excels on both majority and minority classes under the FL
setting?

A possible choice is to re-balance the clients’ local gradients via approaches
such as Seasaw loss [26] or Equalization Loss v2 [27]. But one needs to
customize these methods on different clients with heuristically selected
hyper-parameters, resulting in complicated models with limited generalization
capability. Moreover, the trained local models may have huge variance, as
illustrated in Fig. 4 (a), and hardly formulate a synergistic model to handle
tail class.

In light of the above challenges, this paper proposes Fed-GraB, comprised of a
long-tailed statistic analyzer and a gradient rebalancer, for the Fed-LT
problem, as illustrated in Fig. LABEL:fig:_framework. More specifically, we
develop a federated weight-norm-based method, coined as the Direct Prior
Analyzer (DPA), to derive a prior vector for global data statistics, utilizing
the weight parameters from the global classifier. To overcome the degradation
caused by the distribution discrepancy between the server and clients (or
amongst the clients) in Fed-LT, we establish a self-adjusting gradient
balancer (SGB) based on the estimated global head-tail characteristics, which
is integrated at every client to re-balance the positive and negative
gradients in a class-wise and privacy-preserving manner. A marked advantage of
Fed-GraB is that it trains all heterogeneous clients in a feedback-based
closed-loop manner, encouraging all clients to contribute to the global long-
tailed recognition task collaboratively. We also provide comprehensive
theoretical and empirical analyses to verify the effectiveness of Fed-GraB. We
conduct excessive experiments on multiple benchmark datasets (including the
CIFAR-10/100-LT, ImageNet-LT, and iNaturalist) with both long-tailed and non-
IID data distributions. The results demonstrate that Fed-GraB outperforms
state-of-the-art (SOTA) baselines, including federated optimization for
heterogeneous data, FL methods for LT classification, and centralized LT
learning methods in FL settings.

##  2 Related Work

###  2.1 Federated learning with data heterogeneity

Numerous FL approaches have been developed to cope with the heterogeneous or
imbalanced data distribution. For instance, FedProx [16], FedDyn [17], and
MOON [28] modify the local loss function by adding regularizers at the local
side. SCAFFOLD [18] proposes a control variates-based method to reduce the
clients’ distribution drift brought by the discrepancies. FedIR [4]
dynamically adjusts the sampling ratio of each client based on each client’s
contribution and data distribution situation. And FedNova [20] is proposed to
tackle the induced objective inconsistency problem. On the server side,
FedAvgM [19] and FedAdaGrad [29] are proposed to mitigate the performance
degradation. Moreover, FL-oriented client selection [30, 31] and data
augmentation strategies [32] have been investigated to enhance the
performance. To address the poor generalization of a single generic model,
personalization training methods have been recently explored to maintain
multiple personalized local models [33, 34, 21]. However, most of these
methods pay more attention to the discrepancies in inter-client distributions
while ignoring the inconsistency among different classes, failing to achieve
satisfactory performance on the tail classes. In lieu of this, an alternate
paradigm [35] solely addresses the personalized long-tail conundrum,
disregarding the variances within inter-client distributions. Notable
exceptions could be found in [8, 36, 6] to tackle class imbalance. Still, they
require the exchange of local features [8], private local data distribution
[36] and auxiliary data [6], which would lead to potential privacy issues.

###  2.2 Long-tailed learning

Long-tailed data widely exists in real-world machine learning tasks, where
head classes dominate the training [10, 37]. To tackle the poor performance of
tail classes in such scenarios, many approaches have been explored to boost
the performance of tail classes via class re-balancing, information
augmentation, and module improvements [38, 39, 40, 41, 25]. Re-balancing
strategies, as the mainstream approaches in long-tailed learning, can be
categorized into class re-sampling, re-weighting, and logit adjustments [10,
42]. Re-sampling approaches [43, 44, 45, 46] address the class imbalance via
over-sampling tail classes or under-sampling head classes. On the other hand,
the re-weighting approaches aim to balance the loss( [47, 48, 49, 50]) or
gradients (Seesaw loss [26], Eqlv2 [27]). Specifically, Focal loss [39]
increases the prediction probabilities to achieve better prediction
performance for tail classes. And a label-distribution-aware margin loss LDAM
is introduced by [49] using label frequency. Moreover, a multi-expert
framework (TADE [51]), prototype learning (OLTR [24]), and head-to-tail
knowledge transfer (LEAP [52]) have been proposed for improved long-tailed
data performance. However, such methods’ effectiveness may be constrained in
Fed-LT due to discrepancies in local and global statistics.

##  3 Method

###  3.1 Preliminaries

Federated Learning: Consider an FL system with $N$ clients and an $M$-class
visual recognition dataset $\mathcal{D}=\\{\mathcal{D}_{k}\\}_{k=1}^{N}$ for
classification tasks. $\mathcal{D}_{k}$ denotes the local dataset of client
$k$ with size $|\mathcal{D}_{k}|$. Usually, the FL training process iterates
multiple communication rounds until convergence. In a typical communication
round $l$, the central server randomly selects a subset of clients
$\mathcal{S}_{l}$ and distributes the latest global model $w^{l}$ to them. For
client $k\in\mathcal{S}_{l}$, it performs local update to $w^{l}$ based on
$\mathcal{D}_{k}$, where the locally computed model is denoted as $w^{l}_{k}$.
At the end of round, the global model could be updated by the aggregation
criterion (e.g., FedAvg) for next round computation.

Long-tailed distribution in FL: To characterize the long-tailed data
distribution in Fed-LT, we measure the imbalance factors in global
perspectives, which is denoted by $\text{IF}_{\text{G}}$ and computed over the
global dataset $\mathcal{D}$. We could employ a predefined imbalance
$\text{IF}_{\text{G}}$ to sample from balanced datasets, constructing long-
tailed datasets; see Fig. 2 for visualization.

Positive and Negative Gradients: Notably, positive and negative cumulative
gradients have an important effect on long-tailed tasks such as visual
recognition [53], object detection [27] and instance segmentation [26].
Suppose we have a neural network-based classification model with Cross Entropy
loss as $\mathcal{L}(\bm{z})=-\sum_{i=1}^{M}y_{i}\log(\sigma_{i})$, with
$\sigma_{i}=\frac{e^{z_{i}}}{\sum_{j=1}^{M}e^{z_{j}}}$, where $y_{i}$ is the
ground truth label, $\bm{z}=[z_{1},z_{2},\dots,z_{M}]$ are the logits, and
$\bm{\sigma}=[\sigma_{1},\sigma_{2},\dots,\sigma_{M}]$ denotes the
probabilities of the classifier. Given a sample with label $j$, the positive
gradients are defined as the derivative of the loss with respect to $z_{j}$,
while the rest shall be the negative gradients, i.e.,

$\nabla_{z_{j}}^{\text{pos}}\left(\mathcal{L}\right)=\sigma_{j},\quad\nabla_{z_{i\neq
j}}^{\text{neg}}\left(\mathcal{L}\right)=\sigma_{i}-1.$ (1)

###  3.2 Direct Prior Analyzer (DPA)

Figure 3: The closed-loop in SGB. $\beta_{j}^{pos}(t),\beta_{j}^{neg}(t)$ are
updated according to the controller and the re-weighted gradients are stored
in the collector for future cumulative computation of $\Delta_{j}(t)$.

This section details the proposed DPA module which calculates a prior vector
for balanced training. It utilizes the same information required by FedAvg,
without the need for extra data or distribution details. Essentially, DPA
derives a prior probability vector that discerns potential tail classes by
examining the L2-norm of the weight parameters within the classifiers. And the
reason for this design arises from existing studies [25, 54], revealing that a
model recognizes head or tail classes via classifier and tends to give higher
scores for head classes when trained with imbalanced data. Thus, the weight
norm serves as an effective indicator of the imbalance degree since the norm
of the active weights could empirically reflect the behavior and frequency of
different classes. In particular, let $w^{j}_{g}$ denote the weights of
classifier $j$ in the global model classifier with weights $w_{g}$. The
estimated global distribution would be given by $\mathcal{P}_{c}=\\{p_{j}\mid
h(\|w^{j}_{g}\|_{2})\\}$, where $h(\cdot)$ is a linear transformation for
scaling the norm to share of the total. This estimation can be executed after
global model aggregation with full client participation.

In the Fed-GraB implementation, the prior vector $\mathcal{P}_{c}$ derives
from DPA at the onset of local training. It is also worth mentioning that
$\mathcal{P}_{c}$ does not need to be perfect and only requires information
that reflects the model prediction inclinations at the class level, rather
than a complete correspondence with the distribution itself. See more detailed
analysis in experiments.

###  3.3 Self-adjusting Gradient Balancer for Fed-LT

This part establishes the Self-adjusting Gradient Balancer (SGB), an effective
tool that can be ietegrated into biased classifiers to re-balance local
training and improve the overall performance. Particularly, we consider the
difference between cumulative positive and negative gradients $\Delta(t)$,
with $\Delta_{j}(t)$ for class $j$ defined as
$\Delta_{j}(t)=g^{pos}_{j}(t)-g^{neg}_{j}(t)$, where
$g_{j}(t)=\sum_{i=0}^{t}\nabla_{z_{j}}\left(\mathcal{L}^{t}\right)$ is the
cumulative gradients of class $j$, capturing the overall imbalance degree
throughout the training process. In a toy balanced scenario with $M$ samples
each with equal probability to the $M$ classes, the expected target $z_{j}(t)$
of $\Delta_{j}(t)$ is:

$E(z_{j}(t))=\sum_{i=1}^{M-1}(E(\sigma_{i}(t))-1)-E(\sigma_{j}(t))=0,$ (2)

indicating that $\Delta(t)$ approaches zero when the distributions become
identical.

The core idea of SGB is to consistently align $\Delta_{j}(t)$ to 0 for tail
classes in a closed-loop and self-adjusting manner. Specifically, we carry out
a proportional-integral-derivative controller [55] to continually update the
re-weighting coefficients of gradients with regard to the logits in a class-
wise manner, as demonstrated in Fig. 3. The re-weighting process is based on
adaptive self-adjustment with error feedback until $\Delta(t)$ reaches the
pre-determined balanced target, rather than following heuristic methods as
[26, 27]. The error feedback $e_{j}(t)=\Delta_{j}(t)-z_{j}(t)$ for class $j$
represents the distance between the current status $\Delta_{j}(t)$ and a
target $z_{j}(t)$ during training. Given $e(t)$, the output of the controller
in SGB is

$u_{j}(t)=K_{P}e_{j}(t)+K_{I}\sum_{j=0}^{t}e_{j}(t)+K_{D}(e_{j}(t)-e_{j}(t-1)),$
(3)

where $K_{P}$, $K_{I}$, and $K_{D}$ are controlling factors.

Algorithm 1 Local training process of Fed-GraB

1:$w_{k}^{l}$, local model at round $l$ for client $k$

2:Updated local model $w_{k}^{l+1}$

3:functionClientUpdate($w^{l}_{k}$)

4:Update prior vector $\mathcal{P}_{c}$ by DPA with $w_{k}^{l}$

5:for batch $b$ $\in$ $\mathcal{B}$ do

6: for each classifier $z_{j}$ $\in$ $z$ do

7: #Initialize $\Delta_{j}(t)=0$

8: Compute
$\nabla_{z_{j}}(\mathcal{L};b)=\left[\nabla_{z_{j}}^{pos}(\mathcal{L};b),\nabla_{z_{j}}^{neg}(\mathcal{L};b)\right]$

9: Sample a random value $r\sim\text{Uniform}(0,1)$

10: # Gradient Balancer

11: Input of SGB $e(t)$ $\leftarrow$ $\Delta_{j}(t)$

12: Compute output of SGB $u(t)$ by Eq. 7

13: Compute re-weighting coefficients $\beta_{j}$ by Eq. 4

14: # Gradient Collector

15: Update $\Delta_{j}(t)$ with the re-weighted gradients by Eq. 5

16: $w^{l+1}_{k}\leftarrow
w^{l}_{k}-\eta\sum_{j=0}^{M-1}\beta_{j}\nabla_{z_{j}}(\mathcal{L};b)$

17:return $w^{l+1}_{k}$

The proportional item with $K_{P}$ shows the controller would give bigger re-
weighting coefficients $\beta_{j}$ to compensate the imbalance for current
larger errors. Meanwhile, the integral item with $K_{I}$ could look back from
the past errors to reduce the steady-state offset which could not be handled
by the proportional item. Furthermore, as computing $\beta_{j}$ only with the
proportional and integral loop could hardly achieve steady and smooth re-
weighting, we introduce the third differential item with $K_{D}$ to control
the fluctuations based on future expectations. For instance, when
$\Delta_{j}(t)$ is close to $0$, the differential item would prohibit $u(t)$
from diverging re-weighting and vise versa. Overall, the three items work
together to achieve quick, precise and stable adjustments.

We first map $u_{j}(t)$ through a simple activation function $\phi(\cdot)$ to
calculate $\beta_{j}^{neg}=\phi(-u_{j}(t)),~{}\beta_{j}^{pos}=\phi(u_{j}(t))$,
where $\phi(x)=\frac{\gamma}{1+\delta e^{-\zeta x}}$. Then, we combine the
global information obtained by DPA to calculate the final coefficients for
positive and negative gradients per iteration as:

$\displaystyle\beta_{j}=\mathbb{I}_{r>\mathcal{P}_{c}[j]}\cdot[\beta^{pos}_{j},\beta^{neg}_{j}]+\mathbb{I}_{r\leq\mathcal{P}_{c}[j]}\cdot[1,1],$
(4)

where $\mathbb{I}(\cdot)$ is the indicator function. Consequently, the re-
weighted gradients are computed as:

$\left\\{\begin{matrix}\nabla_{z_{j}}^{\text{pos}^{\prime}}\left(\mathcal{L}^{t}\right)=\beta_{j}^{\text{pos}}(t)\nabla_{z_{j}}^{\text{pos}}\left(\mathcal{L}^{t}\right),\\\
\nabla_{z_{j}}^{\text{neg}^{\prime}}\left(\mathcal{L}^{t}\right)=\beta_{j}^{\text{neg}}(t)\nabla_{z_{j}}^{\text{neg}}\left(\mathcal{L}^{t}\right).\end{matrix}\right.$
(5)

The working mechanism of SGB at a typical client is summarized in Fig 3. As
depicted in Fig LABEL:fig:_framework (a), SGB is implemented across all the
classes in each client, with parameters adjusted in accordance with the prior
vector derived by DPA. By utilizing global information, this approach
encourages all clients to contribute coherently to the global model while
simultaneously rectifying prediction biases arising from long-tailed datasets.
As shown in Fig LABEL:fig:_framework (b), SGB improves the balance between
cumulative positive and negative gradients. See experiments and Supplementary
for further analysis.

###  3.4 Algorithm Summary for Fed-GraB

The training pipeline of our Fed-GraB framework is presented in Algorithm 1,
which consists of two stages. First, DPA calculates the prior vector
$\mathcal{P}_{c}$ after receiving aggregated weights $w^{l}_{k}$. Then, SGB is
applied to all the classes. The local classifier would be dynamically adjusted
with weighted positive and negative gradients via a $\Delta(t)$-based closed-
loop controller in accordance with prior vector $\mathcal{P}_{c}$ derived from
DPA. The combination with SGB and DPA is detailed in Eq. 4.

###  3.5 Privacy discussions of DPA

As the local prior computation is performed at the client side, there would be
a few privacy concerns with the DPA method. It is, however, noteworthy that
the potential privacy issue exists in the general FL frameworks rather than
specific to our proposed DPA method. For instance, gradient inversion [56] can
pose a threat to almost all gradient transmission-based FL methods without any
privacy-preservation techniques. As the privacy issue of FL framework is
beyond the scope of this work, we briefly include the discussion in this
subsection.

##  4 Experiments

###  4.1 Experimental Setup

Baselines: We consider two types of SOTA baselines: (1) FL-oriented methods to
tackle data heterogeneity (FedProx [16], FedNova [20]) or federated long-
tailed data-oriented (FedIR [4] and CReFF [8]), and FedAvg is also included
for reference; (2) Long-tailed learning (LT)-oriented methods ($\tau$-norm
[25], Eqlv2 [27], LDAM [49], Focal-loss [39] and GCL-loss [42]) applied at
local training of each client. We also provide the results of the long-tailed
methods in centralized learning (CL) settings as the oracle performance upper
bound, including SGB.

Datasets: We conduct the experiments in three benchmark datasets for long-
tailed classification, i.e., CIFAR-10/100-LT [57], ImageNet-LT [58].
CIFAR-10/100-LT are sampled into long-tailed distribution by exponential
distribution controlled by IF [49], and we use the same configurations as in
[24] for ImageNet-LT with the number of images per class ranging from 5 to
1280. To evaluate the performance on real-world data, we also conduct
experiments on iNaturalist-User-160k, with 160k examples of 1,023 species
classes and partitioned on the basis of iNaturalist-2017 [59].

Federated settings: We use non-IID data partitions for all experiments,
implemented via symmetric Dirichlet distributions with concentration parameter
$\alpha$ to control the identicalness of local data distributions among all
the clients. We train a ResNet-18 over $N=40$ clients on CIFAR-10-LT.
ResNet-34 and ResNet-50 are used on CIFAR-100-LT and ImageNet-LT respectively
with $N=20$ clients. For iNaturalist-160k, we use the same settings as
ImageNet-LT.

Implementation of baselines: For the sake of fairness, we keep consistent
settings external for experiments. We conduct experiments using a starting
model from FedAvg. For CReFF, the number of federated features is 100, we use
0.1, 0.01 as federated feature learning rate and main net learning rate
respectively on CIFAR-10/100-LT. We report the official result [8] on
ImageNet-LT. As $\tau$-norm is one-shot method, we provide a pre-trained model
by FedAvg and adjust the classifier weights on the server as post-process.
Focal-Loss and EQL v2 do not require distribution prior. We directly replace
the local cross-entropy loss by their proposed re-balancing methods. For LDAM,
we use the local data distribution for its local training.

Setting Method $\text{IF}_{\text{G}}$=10 $\text{IF}_{\text{G}}$=50
$\text{IF}_{\text{G}}$=100 Many Med Few All Many Med Few All Many Med Few All
CL Softmax 0.901 0.879 0.886 $0.893_{\pm 0.003}$ 0.908 0.776 0.742 $0.817_{\pm
0.003}$ 0.920 0.745 0.675 $0.767_{\pm 0.001}$ Eqlv2 0.902 0.880 0.874
$0.890_{\pm 0.002}$ 0.903 0.774 0.774 $0.819_{\pm 0.006}$ 0.912 0.751 0.679
$0.775_{\pm 0.012}$ LDAM 0.957 0.799 0.854 $0.838_{\pm 0.026}$ 0.964 0.739
0.685 $0.753_{\pm 0.011}$ 0.936 0.729 0.610 $0.698_{\pm 0.030}$ SGB-CL 0.897
0.901 0.901 $0.898_{\pm 0.008}$ 0.891 0.817 0.833 $0.846_{\pm 0.004}$ 0.901
0.738 0.814 $0.818_{\pm 0.003}$ $\alpha$=1 FedAvg 0.896 0.858 0.846
$0.877_{\pm 0.001}$ 0.888 0.771 0.693 $0.792_{\pm 0.005}$ 0.922 0.716 0.616
$0.737_{\pm 0.001}$ FedProx 0.898 0.859 0.854 $0.877_{\pm 0.002}$ 0.891 0.773
0.691 $0.794_{\pm 0.002}$ 0.921 0.725 0.582 $0.729_{\pm 0.002}$ FedNova 0.912
0.853 0.848 $0.882_{\pm 0.002}$ 0.903 0.757 0.702 $0.797_{\pm 0.003}$ 0.934
0.734 0.599 $0.739_{\pm 0.005}$ FedIR 0.966 0.823 0.862 $0.868_{\pm 0.001}$
0.972 0.775 0.693 $0.784_{\pm 0.002}$ 0.969 0.755 0.576 $0.728_{\pm 0.004}$
CReFF 0.911 0.850 0.887 $0.884_{\pm 0.002}$ 0.896 0.769 0.664 $0.791_{\pm
0.003}$ 0.935 0.723 0.574 $0.726_{\pm 0.002}$ 2-14[4pt/3pt] $\tau$-norm 0.887
0.871 0.908 $0.884_{\pm 0.003}$ 0.878 0.790 0.725 $0.805_{\pm 0.002}$ 0.922
0.726 0.668 $0.760_{\pm 0.004}$ Eqlv2-FL 0.896 0.852 0.857 $0.878_{\pm 0.003}$
0.886 0.786 0.690 $0.790_{\pm 0.009}$ 0.919 0.704 0.597 $0.729_{\pm 0.003}$
LDAM-FL 0.901 0.845 0.825 $0.863_{\pm 0.004}$ 0.881 0.739 0.662 $0.768_{\pm
0.005}$ 0.891 0.638 0.495 $0.679_{\pm 0.025}$ Focal-FL 0.887 0.839 0.834
$0.869_{\pm 0.005}$ 0.877 0.744 0.665 $0.775_{\pm 0.002}$ 0.916 0.701 0.558
$0.733_{\pm 0.037}$ GCL-FL 0.923 0.747 0.781 $0.796_{\pm 0.000}$ 0.949 0.748
0.689 $0.761_{\pm 0.005}$ 0.963 0.726 0.608 $0.726_{\pm 0.001}$ Fed-GraB 0.886
0.882 0.893 $0.885_{\pm 0.001}$ 0.875 0.784 0.775 $0.818_{\pm 0.003}$ 0.910
0.698 0.713 $0.766_{\pm 0.003}$ $\alpha$=0.5 FedAvg 0.890 0.864 0.861
$0.876_{\pm 0.002}$ 0.865 0.772 0.685 $0.781_{\pm 0.003}$ 0.906 0.720 0.585
$0.719_{\pm 0.005}$ FedProx 0.883 0.864 0.863 $0.874_{\pm 0.000}$ 0.857 0.776
0.688 $0.782_{\pm 0.001}$ 0.892 0.712 0.564 $0.715_{\pm 0.011}$ FedNova 0.903
0.856 0.834 $0.877_{\pm 0.002}$ 0.888 0.773 0.679 $0.788_{\pm 0.003}$ 0.924
0.739 0.609 $0.739_{\pm 0.004}$ FedIR 0.955 0.816 0.870 $0.866_{\pm 0.001}$
0.961 0.771 0.698 $0.781_{\pm 0.001}$ 0.976 0.726 0.562 $0.715_{\pm 0.005}$
CReFF 0.900 0.838 0.880 $0.877_{\pm 0.004}$ 0.878 0.778 0.664 $0.786_{\pm
0.003}$ 0.932 0.699 0.580 $0.718_{\pm 0.003}$ 2-14[4pt/3pt] $\tau$-norm 0.883
0.863 0.880 $0.877_{\pm 0.004}$ 0.867 0.759 0.728 $0.795_{\pm 0.003}$ 0.962
0.726 0.611 $0.731_{\pm 0.011}$ Eqlv2-FL 0.895 0.858 0.855 $0.877_{\pm 0.002}$
0.872 0.771 0.639 $0.775_{\pm 0.005}$ 0.898 0.717 0.586 $0.714_{\pm 0.011}$
LDAM-FL 0.885 0.860 0.848 $0.850_{\pm 0.006}$ 0.863 0.742 0.673 $0.709_{\pm
0.037}$ 0.877 0.583 0.490 $0.657_{\pm 0.024}$ Focal-FL 0.868 0.853 0.858
$0.865_{\pm 0.005}$ 0.842 0.780 0.661 $0.767_{\pm 0.005}$ 0.883 0.703 0.581
$0.728_{\pm 0.029}$ GCL-FL 0.902 0.731 0.813 $0.798_{\pm 0.004}$ 0.938 0.725
0.725 $0.768_{\pm 0.001}$ 0.954 0.730 0.623 $0.713_{\pm 0.016}$ Fed-GraB 0.864
0.891 0.897 $0.881_{\pm 0.002}$ 0.836 0.790 0.783 $0.806_{\pm 0.001}$ 0.910
0.698 0.713 $0.761_{\pm 0.008}$

Table 1: Test accuracies of Fed-GraB/SGB and SOTA methods on CIFAR-10-LT with
diverse imbalanced and heterogeneous data settings. More results (e.g., DPA
with CL re-weighting methods and IID settings) are in the Supplementary.

Method CIFAR-100-LT ImageNet-LT iNaturalist-160k Many Med Few All Many Med Few
All Many Med Few All FedAvg 0.643 0.410 0.182 $0.365_{\pm 0.001}$ 0.428 0.258
0.127 $0.287_{\pm 0.006}$ 0.596 0.425 0.242 $0.434_{\pm 0.002}$ FedProx 0.639
0.416 0.181 $0.366_{\pm 0.000}$ 0.439 0.268 0.128 $0.292_{\pm 0.002}$ 0.582
0.424 0.241 $0.425_{\pm 0.011}$ FedNova 0.664 0.429 0.195 $0.378_{\pm 0.009}$
0.415 0.234 0.114 $0.265_{\pm 0.002}$ 0.564 0.403 0.226 $0.404_{\pm 0.006}$
FedIR 0.634 0.410 0.182 $0.364_{\pm 0.000}$ 0.388 0.206 0.074 $0.236_{\pm
0.012}$ 0.579 0.387 0.191 $0.396_{\pm 0.003}$ CReFF 0.684 0.440 0.146
$0.401_{\pm 0.002}$ - - - $0.263_{\pm 0.000}$ - - - - 1-13[4pt/3pt]
$\tau$-norm 0.459 0.362 0.323 $0.368_{\pm 0.003}$ 0.347 0.285 0.260
$0.288_{\pm 0.018}$ 0.537 0.433 0.287 $0.434_{\pm 0.003}$ Eqlv2-FL 0.652 0.434
0.198 $0.381_{\pm 0.002}$ 0.433 0.262 0.118 $0.281_{\pm 0.006}$ 0.570 0.397
0.376 $0.440_{\pm 0.014}$ LDAM-FL 0.639 0.409 0.168 $0.355_{\pm 0.005}$ 0.365
0.216 0.112 $0.242_{\pm 0.002}$ 0.560 0.409 0.244 $0.414_{\pm 0.003}$ Focal-FL
0.645 0.418 0.179 $0.367_{\pm 0.001}$ 0.424 0.266 0.136 $0.283_{\pm 0.005}$
0.596 0.430 0.247 $0.430_{\pm 0.005}$ GCL-FL 0.567 0.346 0.156 $0.301_{\pm
0.012}$ 0.317 0.209 0.124 $0.224_{\pm 0.001}$ 0.565 0.375 0.196 $0.388_{\pm
0.007}$ Fed-GraB 0.683 0.553 0.221 $0.411_{\pm 0.002}$ 0.407 0.294 0.215
$0.311_{\pm 0.003}$ 0.527 0.449 0.372 $0.451_{\pm 0.004}$

Table 2: Test accuracies of various methods on CIFAR-100-LT, ImageNet-LT and
iNaturalist-160k with non-IID data settings.

###  4.2 Comparison with State-of-the-art Methods

Evaluation on CIFAR-10-LT:

The performance of Fed-GraB/SGB on diverse settings are reported in Tab. 9.
Notably, Fed-GraB achieves the best overall accuracies on all settings, with a
significant improvement on the tail classes while ensuring good performance of
the head classes. The performance gain becomes more evident under extremely
imbalanced data. Moreover, the SGB implemented in the CL setting (i.e.,
performance upper bound testing) surpasses existing SOTA long-tail methods,
further demonstrating the universal effectiveness of Fed-GraB under different
scenarios.

Evaluation on CIFAR-100-LT, ImageNet-LT, and iNaturalist-160k: Tab. 2 shows
that Fed-GraB consistently outperforms all baselines on test overall
accuracies for CIFAR-100-LT and ImageNet-LT, with remarkable improvements on
middle and tail classes. While LT-orientated methods obtain better results
than the FL method in the extremely imbalanced ImageNet-LT, Fed-GraB can still
surpass it111The result of CReFF on ImageNet-LT are from [8] and the result on
iNaturalist is not included due to memory issues in large-scale dataset.. The
reason boils down to that for classification tasks with a large number of
classes, local data samples for each class would be extremely small. In this
case, the SGB with global prior for class-wise re-balancing is more effective
in learning a better aggregated model.

As for iNaturalist-160k, Fed-GraB significantly boosts the classification
accuracy on tail classes while maintaining the best overall performance
compared to FL methods. In comparison to LT-oriented methods, Fed-GraB’s SGB
provides a robust mechanism against data heterogeneity, leading to an
outstanding performance on both 3-shot and overall metrics.

A salient advantage of Fed-GraB is that it achieves universally outperformance
on three large-scale or real-world scenarios, demonstrating the robustness
over other methods such as CReFF, $\tau-norm$ or Eqlv2-FL which attain good
performance on a specific dataset.

Evaluation of Communication Efficiency: Tab. 3 presents the communication
efficiency of different methods trained on CIFAR-10-LT under two non-IID
settings. We compare the required communication rounds for the tail classes to
reach 55% test accuracy. We can see from this table that Fed-GraB attains the
best convergence rate under both settings, confirming its effectiveness on the
tail classes.

###  4.3 Effectiveness of DPA and SGB

SGB for consistent gradient re-weighting across clients: When various clients
employ a gradient re-weighting method on a specific class, the strength of
compensating divergence because of the diverse local distributions and local
dataset sizes. Therefore, although traditional re-weighting methods can
mitigate the imbalance, discrepancies still exist. We demonstrate the
advantage of SGB for addressing this issue. Specifically, we compute the mean
and standard deviation (std) of $\Delta_{j}(t)$ across all clients in
CIFAR-10-LT for both Eqlv2 and Fed-GraB, as illustrated in Fig. 4 (a).

Method |  $\text{IF}_{\text{G}}$=50 |  $\text{IF}_{\text{G}}$=100
---|---|---
$\alpha=1$ | $\alpha=0.5$ | $\alpha=1$ | $\alpha=0.5$
FedAvg | 96 | 122 | 88 | 192
CReFF | 285 | 286 | 309 | 348
Eqlv2-FL | 301 | 210 | 191 | 205
LDAM-FL | 483 | 427 | - | -
Focal-FL | 276 | 174 | 347 | 257
Fed-GBA | 59 | 63 | 95 | 122

Table 3: A comparison of communication efficiency on CIFAR-10, in terms of the
number of required rounds, where “-” indicate the method cannot obtain the
target accuracy.

We make two observations here. First, the mean of $\Delta_{j}(t)$ in Eqlv2
keeps decreasing alongside the training process while Fed-GraB can always
align it to a near-zero value, suggesting that Fed-GraB can better re-weight
the positive and negative gradients. Second, Eqlv2 exhibits a large variance
among clients during training. In contrast, Fed-GraB maintains a very small
deviation, evidently demonstrating the Fed-GraB can consistently motivate the
heterogeneous clients to converge towards a better aggregated global model.

Effectiveness on global distribution and tail class estimation: We evaluate
DPA under three IF settings with two metrics: (1) the L2-distance between the
prior vector $\mathcal{P}_{c}$ and ground truth distribution; (2) the tail
identification accuracy (percentage of tail classes that are identified
correctly by DPA). We plot the two metrics on a global model with FedAvg for
200 rounds in Fig 4 (b). We can see that DPA converges after 70 rounds of
training and captures more than 90% of the tail classes under different IFs.
Moreover, the tail identification accuracy becomes more accurate with a higher
IF, which is attributed to the extreme imbalance among classes (i.e., easier
to distinguish).

|  |
---|---|---
(a) | (b) | (c)
Figure 4: (a): Visualization of mean and std of $\Delta_{j}(t)$ during training; (b): Tail identification performance with DPA; (c): Comparison of accuracies with global/local-SGB mounting strategy. |  |
---|---|---
(a) | (b) | (c)
Figure 5: (a): An illustration of effectiveness of $\mathcal{P}_{c}$ with
class-wise performance; (b): Mounting strategies with SGB; (c): An
illustration of the trend of $\Delta(t)$ with different values of $z(t)$.

Effectiveness of global $\mathcal{P}_{c}$: We investigate a local version
(Local-SGB) of Fed-GraB (Global-SGB), where each client implements SGB based
on their local distribution instead of the global prior vector derived by DPA.
The results on CIFAR-10-LT in Fig. 4 (c) show that Global-SGB achieves a
higher test accuracy. Furthermore, we visualize the accuracies of FedAvg and
Fed-GraB after local updates on the received global model in Fig. 12 (a),
which demonstrate that Fed-GraB can achieve better and more balanced
performance across classes, especially for tails (e.g., class 2, 3, 8, 9).

Mounting strategies with SGB: In Fed-GraB, SGB is universally applied to all
classifiers through the use of DPA, rather than selectively targeting specific
classes without DPA. We demonstrate that a full mounting approach with DPA can
more readily attain optimal performance compared to a custom-tailored tailed
class (i.e., mounting on 7 to 9 classes). As depicted in Fig. 12 (b), the
model excels when SGB is employed on tail classes, while accuracy diminishes
when mounting all classes or solely the exceedingly biased classes. Notably,
DPA significantly bolsters overall performance.

###  4.4 Ablation Study and Model Analysis

The target of $\Delta(t)$: A critical step SGB is to compute re-weighting
coefficients $\beta$ by $e(t)=\Delta(t)-z(t)$. Here we argue that, for a
static $z(t)$, different values of $z(t)$ would not lead to significant
difference in training, as long as those values are not extremely biased. We
visualize the training process with two different $z(t)$ in Fig. 12 (c) by
assuming a continuous scenario. The training would consist of two phases
during the closed-loop controlling. The Phase 1 could be regarded as a noisy
initialization stage which is usually very short, resulting in different
initial values based on $e(t)$. Afterwards, the Phase 2 is started and similar
adjusting behaviors of $e(t)$ could be observed, i.e., $\Delta(t)$ fluctuates
around $z(t)$, leading to similar $\beta$ that stands for re-weighting
strength. As the duration $\delta$ of Phase 1 is negligibly short and would
not impose significant influence for training. The experimental results with
different target values in Fig. 6 (a) further demonstrate our argument. More
analysis is presented in the Supplementary.

|  |
---|---|---
(a) | (b) | (c)
Figure 6: (a): Performance ablation with different $\Delta(t)$; (b): Ablation
on different $K_{P}$, $K_{I}$ and $K_{D}$ in Fed-GraB; (c): Performance
evaluation with diverse $\text{IF}_{\text{G}}$.

Ablation on SGB hyper-parameters: To investigate the effects of hyper-
parameters $K_{P}$, $K_{I}$, and $K_{D}$, we observe the real-time $\Delta(t)$
during the training process with various parameter groups, as depicted in Fig.
6 (b) using the parameter sets: $K_{P}={1,10,10,10}$, $K_{I}={0,0,0,0.01}$,
$K_{D}={0,0,0.1,0.1}$. We notice that $K_{P}=10$ (green) more effectively
constrains $\Delta(t)$ around $0$ compared to $K_{P}=1$ (orange). To reduce
fluctuations, we incorporate $K_{D}=0.1$ (red). In this case, $\Delta(t)$
remains above $0$, necessitating the introduction of $K_{I}$ (purple) to
correct the static deviation. Notably, SGB demonstrates substantial robustness
to the parameter group, as further evidenced in the Supplementary.

Fed-GraB on different imbalance levels: To demonstrate Fed-GraB is versatile
across different global data distributions, we conduct experiments with global
imbalance factors ranging from 5 to 50. As shown in Fig.6 (c), Fed-GraB
achieves better performance than others on a broad range of
$\text{IF}_{\text{G}}$. The results indicate that Fed-GraB could alleviate the
performance degradation in moderate imbalanced cases, while yielding more
significant improvements on highly imbalanced data.

Computational and Storage Cost of SGB:

The additional cost from SGB are mainly attributed to the computation and
storage of $\Delta(t)$ and $u(t)$. As shown in Eq. 7, the main computational
steps are the differential and summation of $e(t)$, which should be linearly
proportional to number of the gradients. Such extra computational cost is
analogous to the additional manipulations of gradients in advanced stochastic
gradient decent methods such as Momentum or Adam [60]. Overall, the extra
computation which is implemented with several lines of code could be done very
quickly. Besides, Fed-GraB needs some extra storage cost to store the weighted
gradients which is quite cheap as well.

##  5 Conclusions and Limitations

We proposed Fed-GraB, a self-adjusting and closed-loop gradient re-balancing
framework for Fed-LT tasks. Fed-GraB comprises DPA, a federated global long-
tailedness analyzer, and SGB, a local gradient balancer, addressing the
discrepancies in inter-client and inter-class statistics. We carried out
extensive experiments to analyze the functionality of different components in
Fed-GraB and demonstrate the efficacy of Fed-GraB in various Fed-LT
configurations.

Limitations. In this work, we mainly focus on aggregating a global Fed-LT
model, while extensions to include clients are of high interest such as
training superior heterogeneous clients with with personalized strategies.
Besides, experiments with real-world medical or autonomous vehicle datasets
could be examined to further demonstrate the effectiveness of Fed-GraB.

## Acknowledgements

This work is supported by the National Natural Science Foundation of China
(Grant No. 62106222, No. 62201504), the Natural Science Foundation of Zhejiang
Province, China(Grant No. LZ23F020008, No. LGJ22F010001), Zhejiang Lab Open
Research Project (No. K2022PD0AB05) and the Zhejiang University-Angelalign
Inc. R$\&$D Center for Intelligent Healthcare.

## References

  * [1] Q. Yang, Y. Liu, T. Chen, and Y. Tong, “Federated machine learning: Concept and applications,” ACM Transactions on Intelligent Systems and Technology (TIST), vol. 10, no. 2, pp. 1–19, 2019.
  * [2] T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, “Federated learning: Challenges, methods, and future directions,” IEEE Signal Processing Magazine, vol. 37, no. 3, pp. 50–60, 2020.
  * [3] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in Artificial intelligence and statistics, pp. 1273–1282, PMLR, 2017.
  * [4] T.-M. H. Hsu, H. Qi, and M. Brown, “Federated visual classification with real-world data distribution,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part X, pp. 76–92, 2020.
  * [5] Q. Li, Y. Diao, Q. Chen, and B. He, “Federated learning on non-iid data silos: An experimental study,” in 2022 IEEE 38th International Conference on Data Engineering (ICDE), pp. 965–978, IEEE, 2022.
  * [6] L. Wang, S. Xu, X. Wang, and Q. Zhu, “Addressing class imbalance in federated learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, pp. 10165–10173, 2021.
  * [7] J. Xu, Z. Chen, T. Q. Quek, and K. F. E. Chong, “Fedcorr: Multi-stage federated learning for label noise correction,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10184–10193, 2022.
  * [8] X. Shang, Y. Lu, G. Huang, and H. Wang, “Federated learning on heterogeneous and long-tailed data via classifier re-training with federated features,” arXiv preprint arXiv:2204.13399, 2022.
  * [9] Z. Chen, S. Liu, H. Wang, H. H. Yang, T. Q. Quek, and Z. Liu, “Towards federated long-tailed learning,” in International Workshop on Trustworthy Federated Learning in Conjunction with IJCAI 2022 (FL-IJCAI’22), 2022\.
  * [10] Y. Zhang, B. Kang, B. Hooi, S. Yan, and J. Feng, “Deep long-tailed learning: A survey,” arXiv preprint arXiv:2110.04596, 2021.
  * [11] L. Yang, H. Jiang, Q. Song, and J. Guo, “A survey on long-tailed visual recognition,” International Journal of Computer Vision, pp. 1–36, 2022\.
  * [12] H. Tang, Z. Li, Z. Peng, and J. Tang, “Blockmix: Meta regularization and self-calibrated inference for metric-based meta-learning,” in ACM Multimedia, pp. 610–618, ACM, 2020.
  * [13] O. Makansi, Ö. Çiçek, Y. Marrakchi, and T. Brox, “On exposing the challenging long tail in future prediction of traffic actors,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13147–13157, 2021.
  * [14] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11621–11631, 2020.
  * [15] A. Haque, A. Milstein, and L. Fei-Fei, “Illuminating the dark spaces of healthcare with ambient intelligence,” Nature, vol. 585, no. 7824, pp. 193–202, 2020.
  * [16] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated optimization in heterogeneous networks,” Proceedings of Machine Learning and Systems, vol. 2, pp. 429–450, 2020.
  * [17] D. A. E. Acar, Y. Zhao, R. Matas, M. Mattina, P. Whatmough, and V. Saligrama, “Federated learning based on dynamic regularization,” in International Conference on Learning Representations, 2021.
  * [18] S. P. Karimireddy, S. Kale, M. Mohri, S. Reddi, S. Stich, and A. T. Suresh, “Scaffold: Stochastic controlled averaging for federated learning,” in International Conference on Machine Learning, pp. 5132–5143, PMLR, 2020.
  * [19] T.-M. H. Hsu, H. Qi, and M. Brown, “Measuring the effects of non-identical data distribution for federated visual classification,” arXiv preprint arXiv:1909.06335, 2019.
  * [20] J. Wang, Q. Liu, H. Liang, G. Joshi, and H. V. Poor, “Tackling the objective inconsistency problem in heterogeneous federated optimization,” Advances in neural information processing systems, vol. 33, pp. 7611–7623, 2020\.
  * [21] A. Z. Tan, H. Yu, L. Cui, and Q. Yang, “Towards personalized federated learning,” IEEE Transactions on Neural Networks and Learning Systems, 2022\.
  * [22] M. Luo, F. Chen, D. Hu, Y. Zhang, J. Liang, and J. Feng, “No fear of heterogeneity: Classifier calibration for federated learning with non-iid data,” Advances in Neural Information Processing Systems, vol. 34, pp. 5972–5984, 2021.
  * [23] J. Zhang, Z. Li, B. Li, J. Xu, S. Wu, S. Ding, and C. Wu, “Federated learning with label distribution skew via logits calibration,” in International Conference on Machine Learning, pp. 26311–26329, PMLR, 2022.
  * [24] Z. Liu, Z. Miao, X. Zhan, J. Wang, B. Gong, and S. X. Yu, “Large-scale long-tailed recognition in an open world,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2537–2546, 2019.
  * [25] B. Kang, S. Xie, M. Rohrbach, Z. Yan, A. Gordo, J. Feng, and Y. Kalantidis, “Decoupling representation and classifier for long-tailed recognition,” in International Conference on Learning Representations, 2019.
  * [26] J. Wang, W. Zhang, Y. Zang, Y. Cao, J. Pang, T. Gong, K. Chen, Z. Liu, C. C. Loy, and D. Lin, “Seesaw loss for long-tailed instance segmentation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9695–9704, 2021.
  * [27] J. Tan, X. Lu, G. Zhang, C. Yin, and Q. Li, “Equalization loss v2: A new gradient balance approach for long-tailed object detection,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1685–1694, 2021.
  * [28] Q. Li, B. He, and D. Song, “Model-contrastive federated learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10713–10722, 2021.
  * [29] S. Reddi, Z. Charles, M. Zaheer, Z. Garrett, K. Rush, J. Konečnỳ, S. Kumar, and H. B. McMahan, “Adaptive federated optimization,” arXiv preprint arXiv:2003.00295, 2020.
  * [30] R. Balakrishnan, T. Li, T. Zhou, N. Himayat, V. Smith, and J. Bilmes, “Diverse client selection for federated learning: Submodularity and convergence analysis,” in ICML 2021 International Workshop on Federated Learning for User Privacy and Data Confidentiality, 2021.
  * [31] Z. Chen, K. F. E. Chong, and T. Q. Quek, “Dynamic attention-based communication-efficient federated learning,” in International Workshop on Federated and Transfer Learning for Data Sparsity and Confidentiality in Conjunction with IJCAI (FTL-IJCAI’2021), 2021.
  * [32] T. Yoon, S. Shin, S. J. Hwang, and E. Yang, “Fedmix: Approximation of mixup under mean augmented federated learning,” in International Conference on Learning Representations, 2020.
  * [33] A. Li, J. Sun, B. Wang, L. Duan, S. Li, Y. Chen, and H. Li, “Lotteryfl: Personalized and communication-efficient federated learning with lottery ticket hypothesis on non-iid datasets,” arXiv preprint arXiv:2008.03371, 2020.
  * [34] C. T Dinh, N. Tran, and J. Nguyen, “Personalized federated learning with moreau envelopes,” Advances in Neural Information Processing Systems, vol. 33, pp. 21394–21405, 2020.
  * [35] Y. Lu, P. Qian, G. Huang, and H. Wang, “Personalized federated learning on long-tailed data via adversarial feature augmentation,” arXiv preprint arXiv:2303.15168, 2023.
  * [36] M. Duan, D. Liu, X. Chen, R. Liu, Y. Tan, and L. Liang, “Self-balancing federated learning with global imbalanced data in mobile systems,” IEEE Transactions on Parallel and Distributed Systems, vol. 32, no. 1, pp. 59–71, 2020.
  * [37] H. Tang, C. Yuan, Z. Li, and J. Tang, “Learning attention-guided pyramidal features for few-shot fine-grained recognition,” Pattern Recognit., vol. 130, p. 108792, 2022.
  * [38] T. Wang, Y. Li, B. Kang, J. Li, J. Liew, S. Tang, S. Hoi, and J. Feng, “The devil is in classification: A simple framework for long-tail instance segmentation,” in European conference on computer vision, pp. 728–744, Springer, 2020.
  * [39] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, “Focal loss for dense object detection,” in Proceedings of the IEEE international conference on computer vision, pp. 2980–2988, 2017.
  * [40] S. Li, K. Gong, C. H. Liu, Y. Wang, F. Qiao, and X. Cheng, “Metasaug: Meta semantic augmentation for long-tailed visual recognition,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5212–5221, 2021.
  * [41] J. Wang, T. Lukasiewicz, X. Hu, J. Cai, and Z. Xu, “Rsg: A simple but effective module for learning imbalanced datasets,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3784–3793, 2021.
  * [42] M. Li, Y.-m. Cheung, and Y. Lu, “Long-tailed visual recognition via gaussian clouded logit adjustment,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6929–6938, 2022.
  * [43] Y. Zang, C. Huang, and C. C. Loy, “Fasa: Feature augmentation and sampling adaptation for long-tailed instance segmentation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3457–3466, 2021\.
  * [44] B. Liu, H. Li, H. Kang, G. Hua, and N. Vasconcelos, “Gistnet: a geometric structure transfer network for long-tailed recognition,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8209–8218, 2021\.
  * [45] J. Cai, Y. Wang, and J.-N. Hwang, “Ace: Ally complementary experts for solving long-tailed recognition in one-shot,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 112–121, 2021.
  * [46] H. Guo and S. Wang, “Long-tailed multi-label visual recognition by collaborative training on uniform and re-balanced samplings,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15089–15098, 2021.
  * [47] B. Li, Y. Yao, J. Tan, G. Zhang, F. Yu, J. Lu, and Y. Luo, “Equalized focal loss for dense long-tailed object detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6990–6999, 2022.
  * [48] T. Wu, Q. Huang, Z. Liu, Y. Wang, and D. Lin, “Distribution-balanced loss for multi-label classification in long-tailed datasets,” in European Conference on Computer Vision, pp. 162–178, Springer, 2020.
  * [49] K. Cao, C. Wei, A. Gaidon, N. Arechiga, and T. Ma, “Learning imbalanced datasets with label-distribution-aware margin loss,” Advances in neural information processing systems, vol. 32, 2019.
  * [50] T. Wang, Y. Zhu, Y. Chen, C. Zhao, B. Yu, J. Wang, and M. Tang, “C2am loss: Chasing a better decision boundary for long-tail object detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6980–6989, 2022.
  * [51] Y. Zhang, B. Hooi, L. Hong, and J. Feng, “Test-agnostic long-tailed recognition by test-time aggregating diverse experts with self-supervision,” arXiv preprint arXiv:2107.09249, 2021.
  * [52] J. Liu, Y. Sun, C. Han, Z. Dou, and W. Li, “Deep representation learning on long-tailed data: A learnable embedding augmentation perspective,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2970–2979, 2020.
  * [53] J. Tan, C. Wang, B. Li, Q. Li, W. Ouyang, C. Yin, and J. Yan, “Equalization loss for long-tailed object recognition,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11662–11671, 2020.
  * [54] Y. Li, T. Wang, B. Kang, S. Tang, C. Wang, J. Li, and J. Feng, “Overcoming classifier imbalance for long-tail object detection with balanced group softmax,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10991–11000, 2020.
  * [55] S. Bennett, “Development of the pid controller,” IEEE Control Systems Magazine, vol. 13, no. 6, pp. 58–62, 1993.
  * [56] Y. Huang, S. Gupta, Z. Song, K. Li, and S. Arora, “Evaluating gradient inversion attacks and defenses in federated learning,” Advances in Neural Information Processing Systems, vol. 34, pp. 7232–7241, 2021.
  * [57] A. Krizhevsky, G. Hinton, et al., “Learning multiple layers of features from tiny images,” 2009.
  * [58] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE conference on computer vision and pattern recognition, pp. 248–255, Ieee, 2009.
  * [59] G. Van Horn, O. Mac Aodha, Y. Song, Y. Cui, C. Sun, A. Shepard, H. Adam, P. Perona, and S. Belongie, “The inaturalist species classification and detection dataset,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769–8778, 2018.
  * [60] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in ICLR (Poster), 2015.
  * [61] Y. Yang, S. Chen, X. Li, L. Xie, Z. Lin, and D. Tao, “Inducing neural collapse in imbalanced learning: Do we really need a learnable classifier at the end of deep neural network?,” 2022.
  * [62] L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.,” Journal of machine learning research, vol. 9, no. 11, 2008.

Supplementary Material for
Fed-GraB: Federated Long-tailed Learning with Self-Adjusting Gradient Balancer

##  Appendix A Outline

As the supplementary material for our paper titled “Fed-GraB: Federated Long-
tailed Learning with Self-Adjusting Gradient Balancer", we provide further
details about the settings of Fed-LT, implementation detail, additional
experiments as well as the further analysis, organized into following
sections:

  * •

Sec. B introduces detailed settings of Fed-LT, with regard to local/global
imbalance levels, categorization methods and the data partition approaches.

    * –

Sec. B.1 provides a definition of $\text{IF}_{\text{G}}$, a parameter for
long-tailed degree control

    * –

Sec. B.2 explained the motivations and the implementation details of the
categorization methods based on cumulative frequency.

    * –

Sec. B.3 gives the details of dataset partition with addressing the non-IID
and long-tailed data distributions.

  * •

Sec. C gives detailed implementation of Fed-GraB.

  * •

Additional Experiments

    * –

Sec. D.1 shows the performance when DPA is combined with other centralized LT-
oriented re-weighting methods, and further shows the outperformance of our
proposed SGB.

    * –

Sec. D.2 explains the effects of the inner parameters of PID algorithm and
demonstrates its superior dynamic performance.

    * –

Sec. D.3 provides additional ablation study of parameters $K_{P},K_{I},K_{D}$
in the self-adjusting controller.

    * –

Sec. D.4 gives additional ablation of hyper-parameters $\phi,\gamma,\zeta$ in
mapping function.

    * –

Sec. D.5 provides additional results to demonstrates the effectiveness of Fed-
GraB, e.g., IID CIFAR-10-LT, class-wise performance.

  * •

Additional Analysis

    * –

Sec. E.1 discusses the motivation of using static $\Delta_{j}(t)$.

    * –

Sec. E.2 demonstrates that SGB has the capability to be flexibly mounted on a
customized tail class for performance boosting.

    * –

Sec. E.3 shows the effectiveness of DPA in full-scale category sections.

    * –

Sec. E.4 evaluates the performance of different mounting strategies in Fed-
GraB.

    * –

Sec. E.5 demonstrates the the tracking performance of diverse clients.

    * –

Sec. E.6 provides the T-SNE visualization results to further demonstrate the
effectiveness of Fed-GraB.

##  Appendix B Detailed Settings in Fed-LT

###  B.1 Long-tailed distribution in FL

In this part, we give the definitions of $\text{IF}_{\text{L}}^{(k)}$ and
$\text{IF}_{\text{G}}$ to facilitate the formulation of the imbalance level
from the global and local perspectives in Fed-LT.

$\mathcal{D}_{k}$ denotes the local dataset of client $k$ with size
$n_{k}=|\mathcal{D}_{k}|=\sum_{i=1}^{M}n_{k}^{(i)}$, where $n_{k}^{(i)}$ is
the number of data samples of class $i$ in client $k$. To characterize the
long-tailed data distribution in Fed-LT, we use the common assumption in long-
tailed learning, in which each class is sorted according to the number of
images in decreasing order. We shall use the sorted class to obtain class
categories (i.e., head, middle, and tail classes). And we measure the
imbalance factor in both local and global perspectives, given by

$\text{IF}_{\text{L}}^{(k)}=\frac{\max_{j}\\{n_{k}^{(j)}\\}}{\min_{s}\\{n_{k}^{(s)}\\}},\text{IF}_{\text{G}}=\frac{\max_{j}\\{\sum_{i=1}^{N}n_{i}^{(j)}\\}}{\min_{s}\\{\sum_{i=1}^{N}n_{i}^{(s)}\\}},$
(6)

in which $\text{IF}_{\text{L}}^{(k)}$ represents the imbalance factor at
client $k$ and $\text{IF}_{\text{G}}$ is global imbalance factor.

In this work, we mainly investigate the scenarios with diverse
$\text{IF}_{\text{G}}$ and heterogeneous data.

###  B.2 Class categorization via cumulative frequency in Fed-LT

Figure 7: A demonstration of categorization in a cumulative manner. All data
are from one client sampled randomly on CIFAR-10-LT, with
$\text{IF}_{\text{F}}=100$ and $\alpha=0.1$ Figure 8: A depiction of our
categorization in different imbalanced factors $100,50$ and $10$ on
CIFAR-10-LT.

In FL, the data heterogeneity would result in discrepancies in local data
statistics among $\\{\mathcal{D}_{k}\\}_{k=1}^{N}$, especially the diverse
dataset sizes, across different clients, which brings challenges to
categorizing the classes via the number of data samples in each class.
Therefore, we adopt the cumulative frequency in class partition criterion to
categorize head, middle and tail classes for each client.

The categorization process is as follows. Firstly, all the classes are sorted
according to the number of data samples from largest to smallest and then we
can obtain the frequency information sequentially after normalization. After
the above operations, each class is labeled with its frequency value. Then we
set up two thresholds (e.g., 75%, 95%) to partition the classes into head
class $j\in\mathcal{J}$, middle class $m\in\mathcal{M}$ and tail class
$r\in\mathcal{R}$. We visualize our categorization in Fig. 7.

Conventional class categorization methods in CL are mainly based on the number
of data samples in each class. However, in Fed-LT, the head/tail class
categorization could be different due to the discrepancy in local data
statistics, among $\mathcal{D}_{i}$ and $\mathcal{D}_{j}$ (for $i\neq j$). And
the imbalanced local dataset sizes and the diverse local imbalance factor
$\text{IF}_{\text{L}}^{k}$ would make the number-based categorization approach
inappropriate. For example, some clients may not have head classes in some
cases where the maximum number is smaller than the head class threshold. In
contrast, the cumulative frequency-based categorization approach is not
sensitive, as the categorization is based on the frequency or ratio, rather
than an absolute criterion on numbers. The given examples are shown in Fig. 8.
The distributions come from randomly sampled clients in our experiments of
CIFAR-100-LT, when $\text{IF}_{\text{G}}$ is 100, 50, and 10 respectively. The
pink shade stands for tail classes. The categorization results demonstrate the
adopted cumulative frequency-based categorization approach is agnostic to
diverse imbalance factors.

###  B.3 Details of dataset partition

To emulate distributions of Fed-LT (long-tailed at the global level and
heterogeneous among clients), we first obtain global long-tailed datasets and
then divide the long-tailed dataset into
$\mathcal{D}=\\{\mathcal{D}_{k}\\}_{k=1}^{N}$ via dataset partition/sampling
(e.g., non-IID or IID data sampling), where $\mathcal{D}_{k}$ denotes the
local dataset of client $k$ with size $|\mathcal{D}_{k}|$.

For CIFAR-10/100-LT, to shape the original CIFAR-10/100 dataset into long-
tailed distribution, the dataset is truncated followed by an exponential
distribution class-wisely, and the imbalanced level is controlled by the
global imbalanced factor $\text{IF}_{\text{G}}$. After obtaining the truncated
long-tailed dataset, we then conduct sampling to divide $\mathcal{D}$ into
$\\{\mathcal{D}_{k}\\}_{k=1}^{N}$. For non-IID data partition, we use
Dirichlet distribution-based sampling approach [19], and the identicalness
could be controlled by the parameter $\alpha$, where a smaller $\alpha$ would
lead to a more non-IID data distributions. Therefore, followed by this two-
step data partition approach, we can obtain the distributions with global
long-tailed distribution with data heterogeneity across clients.

For ImageNet-LT, long-tail naturally exists via configuration [24]. Based on
the long-tail partition with the number of images per class ranging from 5 to
1280, we further divide the data by Dirichlet Distribution and assign the data
samples to clients.

For iNaturalist, the existing iNaturalist-User-120k[4] is designed for
federated learning contains balanced samples, and it is not appropriate to be
directly used in our experiments.Therefore, a new version is made, namely
iNatualist-120k, with 160k examples of 1,023 species classes and partitioned
based on iNaturalist-2017 [59]. We randomly sample 1023 items from class-100
to class-2300 sorted from head to tail, from which 30 samples per class are
drawn to form the test set, and others belong to the training set with the
number of images per class ranging from 15 to 782.

##  Appendix C Details of implementations

In this section, we provide the details of the controller in SGB and the class
activation approach.

The implementation of SGB. The expression of SGB is:

$u_{j}(t)=K_{p}e_{j}(t)+K_{I}\int_{0}^{t}e_{j}(t)\mathrm{d}t+K_{D}\frac{\mathrm{d}e_{j}(t)}{\mathrm{d}t},$
(7)

However, in Fed-LT, we compute the re-weighting coefficient $\beta_{j}(t)$ for
each mini-batch, so it is infeasible to execute SGB in this discrete AI
system. The model is discretized for the convenience of implementation:

$u_{j}(t)=K_{P}e_{j}(t)+K_{I}\sum_{j=0}^{t}e_{j}(t)+K_{D}(e_{j}(t)-e_{j}(t-1)),$
(8)

In Fed-GraB, we further differentiate Eq. 8:

$\begin{split}\Delta u_{j}(t)&=K_{P}(e_{j}(t)-e_{j}(t-1))+K_{I}e_{j}(t)\\\
&+K_{D}(e_{j}(t)-2e_{j}(t-1)+e_{j}(t-2)),\end{split}$ (9)

The increment of SGB output $\Delta u_{j}(t)$ is added to last $\Delta
u_{j}(t-1)$ calculate a value in current iteration:

$u_{j}(t)=u_{j}(t-1)+\Delta u_{j}(t)$ (10)

Then we utilize $u_{j}(t)$ to generate the re-weighting coefficient $\beta$.
Note that all experiments are conducted by Eq. 9 for convenience, although Eq.
8 can work as well.

Class Activation Considering an exceptional case, in terms of debt classifier
(classifier without positive sample), it will receive no positive sample from
the training set. The continued influx of negative gradients makes any balance
strategy invalid to reach gradient equilibrium. To overcome it, we proposed a
Class Activation strategy. SGB does not start to work until it perceives a
passing positive gradient. We visualize the trends of debt classifier in Fig.
14 (See clients 6, 8, 10 for example.).

Evaluation metrics In our experiments, we compare test accuracy and the
required number of communication rounds to measure the generalization and
communication efficiency performance, respectively. For the test accuracy, we
report the best overall test accuracy for all classes, then we provide the
corresponding test accuracy for the head (many-shot), middle (medium-shot),
and tail (few-shot) classes separately. Here, we define communication
efficiency as the required communication rounds to reach a target accuracy.
Informally, a smaller number of required rounds indicates better communication
efficiency.

Implementation details All experiments were conducted on the PyTorch platform
utilizing the NVIDIA GeForce RTX 3090. To introduce diverse levels of data
heterogeneity and imbalanced global data, we employed different settings for
the parameters $\alpha$ and $\text{IF}_{\text{G}}$. Specifically, in the
CIFAR-10-LT dataset, we used $\alpha=1$ and $\text{IF}_{\text{G}}$ values of
$10,50,100$. In the CIFAR-100-LT dataset, we used $\alpha=0.5$ and a fixed
$\text{IF}_{\text{G}}$ value of $100$. For the ImageNet-LT and iNaturalist
datasets, we used $\alpha=0.5$, $0.1$, and $0.5$, respectively. Regarding the
local training process, we set the local epoch to 5 for CIFAR-10/100-LT and 1
for ImageNet-LT/iNaturalist-160k. The batch size was set to 10 for
CIFAR-10/100-LT and 128 for ImageNet-LT/iNaturalist-160k. The client size
varied depending on the dataset, with 40 clients for CIFAR-10-LT and 20
clients for the other datasets. For local training on CIFAR-10/100-LT, we
utilized ResNet18 and ResNet34, while ResNet50 was employed for the real-world
datasets. We derived the prior vector by DPA after running $80$ rounds with
FedAvg and then update the prior vector via DPA before starting local
training. We report the test accuracies for all classes and the
head/middle/tail classes in our experiments.

##  Appendix D Additional Experiments

###  D.1 Apply DPA to other re-weighting method in Fed-LT

To demonstrate the effectiveness of DPA, we combine it with other
distribution-based re-weighting methods. Note that Focal Loss and EQL v2
conduct re-weighting by internal mechanisms, e.g., independent loss function
and inner gradients. The global versions of these methods involve collecting
clients’ information. We introduce ETF[61] as a new baseline not covered in
the main paper. When training with DPA, the SGB is applied to all classes and
performed using a prior derived from DPA. In contrast, local DPA uses the
local real distribution as a prior. The results are shown in Tab. 4. To
further investigate the scenarios with diverse heterogeneous data, LDAM
utilizes the global distribution as prior knowledge. In this part, we evaluate
the performance of LDAM+DPA on CIFAR-10-LT with fixed
$\text{IF}_{\text{F}}=100$. The results are shown in Tab. 5. These experiments
indicate global prior derived by DPA generally improves performance.
Nevertheless, the global version of other baselines could not eliminate
compensating divergence, leading to worse performance than Fed-GraB. In
summary, we further demonstrate the effectiveness of DPA and the
outperformance of SGB in our proposed Fed-GraB.

Method DPA Many Med Few All w/o w/ $\tau$-norm $\checkmark$ 0.965 0.688 0.577
$0.713_{\pm 0.012}$ $\checkmark$ 0.962 0.726 0.611 $0.731_{\pm 0.011}$ Eqlv2
$\checkmark$ 0.967 0.731 0.567 $0.714_{\pm 0.011}$ $\checkmark$ 0.898 0.717
0.586 $0.721_{\pm 0.006}$ 1-7[4pt/3pt] GCL $\checkmark$ 0.954 0.730 0.623
$0.713_{\pm 0.016}$ $\checkmark$ 0.958 0.722 0.618 $0.730_{\pm 0.003}$ ETF - -
0.499 0.624 0.703 $0.631_{\pm 0.001}$ FedIR - - 0.976 0.726 0.562 $0.715_{\pm
0.005}$ $\checkmark$ 0.953 0.753 0.620 $0.743_{\pm 0.005}$ Fed-GraB
$\checkmark$ 0.910 0.698 0.713 $0.761_{\pm 0.008}$

Table 4: Additional results on CIFAR-10 ($\text{IF}_{\text{G}}$=100,
$\alpha=0.5$).

Setting Method Many Med Few All $\alpha$=0.5 LDAM 0.877 0.583 0.490 0.634
LDAM+DPA 0.955 0.701 0.602 0.713 Fed-GraB 0.887 0.675 0.714 0.754 $\alpha$=1
LDAM 0.891 0.638 0.495 0.657 LDAM+DPA 0.971 0.730 0.616 0.733 Fed-GraB 0.910
0.698 0.713 0.768

Table 5: A comparison of test accuracies for LDAM and DPA on CIFAR-10 with
imbalance factor 100.

###  D.2 Explanation and ablation study for PID

The main objective of Fed-GraB is to drive $\Delta(t)$ towards zero, a
strategy that effectively counteracts imbalanced training. To achieve this, we
utilize a gradient controller, specifically a Proportional-Integral-Derivative
(PID) to adjust $\Delta(t)$.

($K_{p}$, $K_{I}$, $K_{D}$), parameters of PID, are designed for fast
settling, steady state error elimination, and oscillation suppression in the
control process, which affects the _peak time_ ($t_{p}$), _steady state error_
($ess(\infty)$), and _overshoot_ ($\sigma(\%)$) (the smaller, the better),
respectively. _Peak time_ ($t_{p}$) refers to the time it takes for the system
response to reach its maximum (or peak) value for the first time. _Steady
state error_ ($ess(\infty)$) is the difference between the desired final
output zero and the actual output of the system $\Delta(t)$ as the time
approaches infinity. _Overshoot_ ($\sigma(\%)$) is the extent to which the
system’s response exceeds its final steady-state value. An increase in $K_{p}$
shortens $t_{p}$ with overshoot and oscillation, which can be relieved by the
item $K_{D}$, and $K_{I}$ is used to decrease the steady-state error. The
specific values of these parameters are generally tuned empirically, ensuring
that $\Delta(t)$ is rapidly, stably, and precisely confined around zero. The
experiments in Tab. 6 show the effect of $K_{p}$, $K_{I}$, $K_{D}$
individually via corresponding three indicators.

Indeed, the hyper-parameters of SGB require extra tuning. But once they are
set with good performance, the algorithm shall have stable performance across
diverse tasks without additional adjustment. Such an insensitive effect could
be observed in Tab. 1, 2 in the main paper and Tab. 3, 4 in $\mathcal{SM}$.

Notably, our design of the PID controller focuses on minimizing $\Delta(t)$ to
the greatest extent possible, while the overall accuracy of the neural system
demonstrates a high level of robustness even in the presence of slight
variations in parameters. Consequently, alternative controllers can be
employed as substitutes for the PID controller.

$K_{P}$ $K_{I}$ $K_{D}$ $t_{p}$ $ess(\infty)$ $\sigma\%$ Accuracy 0 0 0 - - -
$0.722_{\pm 0.005}$ 3 0 0 $\infty$ 2.3363 - $0.733_{\pm 0.009}$ 10 0 0 334
0.6012 - $0.767_{\pm 0.005}$ 10 0 0.1 461 0.6439 - $0.760_{\pm 0.006}$ 10 0.01
0 276 0.2793 1.1954 $0.778_{\pm 0.005}$ 10 0.01 0.1 317 0.2651 0.2696
$0.764_{\pm 0.007}$

Table 6: Ablation study of PID on CIFAR-10 ($\text{IF}_{\text{G}}$=100,
$\alpha=0.5$).

###  D.3 Additional ablation of $K_{P},K_{I},K_{D}$

$K_{P},K_{I},K_{D}$ are core coefficients in SGB, which can affect the
tracking of $\Delta_{j}(t)$. We have visualized the trends of parameter sets.
To further verify the robustness of SGB, we carry out experiments on
CIFAR-10-LT. We fix $\alpha=1$ and changed $\text{IF}_{\text{G}}$ from 10 to
100, and train for 120 rounds. The results are reported in Tab. 7. The sets of
$\\{K_{P},K_{I},K_{D}\\}$ are $\mathbf{k}_{0}:\\{10,0.01,0.1\\}$,
$\mathbf{k}_{1}:\\{10,0,0.3\\}$, $\mathbf{k}_{2}:\\{10,0.5,0.1\\}$ and
$\mathbf{k}_{3}:\\{5,0.02,0.3\\}$. The experiment result shows that SGB is
parameter-insensitive in Fed-LT for the parameters $K_{P},K_{I},K_{D}$.

###  D.4 Additional ablation of $\gamma,\delta,\zeta$

The $u_{j}(t)$ of SGB is supposed to be enlarged and smoothed out by non-
linear function $\phi(x)=\frac{\gamma}{1+\delta e^{-\zeta x}}$. After the
mapping function, the output is utilized to re-weight gradients. To verify
that our model is not sensitive to the parameters of the mapping function, we
design 3 sets of $\gamma,\delta,\zeta$: namely $\phi_{0}:\\{10,9,-0.5\\}$,
$\phi_{1}:\\{5.5,4.5,-0.5\\}$ and $\phi_{2}:\\{10,9,-0.7\\}$. The mapping
curves are shown in Fig. 9. We give the results for the above parameter sets
in Tab. 8. Mapping functions can be designed as any other forms (e.g. linear
function) in the condition of $\phi(0)=1$.

Paras |  $\text{IF}_{\text{G}}$=100 |  $\text{IF}_{\text{G}}$=50 |  $\text{IF}_{\text{G}}$=10
---|---|---|---
Many | Med | Few | All | Many | Med | Few | All | Many | Med | Few | All
$\mathbf{k}_{0}$ | 0.969 | 0.726 | 0.755 | 0.786 | 0.963 | 0.771 | 0.823 | 0.830 | 0.937 | 0.787 | 0.920 | 0.870
$\mathbf{k}_{1}$ | 0.967 | 0.739 | 0.753 | 0.790 | 0.968 | 0.772 | 0.818 | 0.830 | 0.940 | 0.777 | 0.925 | 0.869
$\mathbf{k}_{2}$ | 0.967 | 0.719 | 0.752 | 0.782 | 0.968 | 0.769 | 0.810 | 0.826 | 0.937 | 0.794 | 0.923 | 0.875
$\mathbf{k}_{3}$ | 0.967 | 0.705 | 0.773 | 0.785 | 0.966 | 0.773 | 0.815 | 0.829 | 0.940 | 0.777 | 0.927 | 0.870

Table 7: Performance comparison of Fed-GraB with different
$K_{P},K_{I},K_{D}$. Figure 9: The mapping functions of SGB with different
parameters.

Setting | Paras |  $\text{IF}_{\text{G}}$=100 |  $\text{IF}_{\text{G}}$=50
---|---|---|---
Many | Med | Few | All | Many | Med | Few | All
$\alpha$=1 | $\phi_{0}$ | 0.963 | 0.685 | 0.758 | 0.770 | 0.946 | 0.708 | 0.828 | 0.804
$\phi_{1}$ | 0.938 | 0.780 | 0.692 | 0.777 | 0.942 | 0.798 | 0.743 | 0.805
$\phi_{2}$ | 0.948 | 0.778 | 0.676 | 0.771 | 0.939 | 0.794 | 0.738 | 0.800
$\alpha$=0.5 | $\phi_{0}$ | 0.963 | 0.673 | 0.739 | 0.758 | 0.948 | 0.709 | 0.832 | 0.806
$\phi_{1}$ | 0.940 | 0.767 | 0.677 | 0.766 | 0.934 | 0.784 | 0.774 | 0.810
$\phi_{2}$ | 0.936 | 0.758 | 0.685 | 0.765 | 0.931 | 0.780 | 0.759 | 0.802

Table 8: Performance comparison of Fed-GraB with different $\gamma,\delta,\zeta$. |
---|---
(a) | (b)
Figure 10: Performance comparison on class-wise accuracy before and after local training on CIFAR-10-LT. Each one stands for a randomly selected client. |
---|---
(a) | (b)
Figure 11: A comparison of the test accuracies of two gradient re-balancing
methods: Local-SGB and Global-SGB, on CIFAR-10-LT with different imbalanced
factors.

###  D.5 Further evaluations of the results in the main paper

As reported in the main paper, our experiments are all implemented with non-
IID data. To better demonstrate the effectiveness of our proposed Fed-GraB on
diverse settings. In Tab. 9, we provide comparison results on CIFAR-10-LT with
IID setting, where the LT dataset $\mathcal{D}$ will be uniformly sampled into
$N$ clients at random. We use the same configuration as the main paper
reported.

As supplements to the main paper, we append the class-wise performance before
and after local training, and the results are shown in Fig. 10. Fed-GraB can
effectively eliminate local tailed performance degradation. We additionally
compare Fed-GraB (Global-SGB) to the Local-SGB where each client mounts SGB
according to local distribution rather than global prior vector derived from
DPA with different $\text{IF}_{\text{G}}$. The results on CIFAR-10-LT in Fig.
11 show that Global-SGB outperforms Local-SGB.

Setting Models $\text{IF}_{\text{G}}$=50 $\text{IF}_{\text{G}}$=100 Many Med
Few All Many Med Few All IID FedAvg 0.894 0.764 0.661 0.785 0.929 0.720 0.595
0.733 FedProx 0.899 0.767 0.660 0.788 0.934 0.729 0.602 0.740 Creff 0.908
0.794 0.668 0.802 0.938 0.734 0.592 0.738 FedNova 0.904 0.786 0.685 0.803
0.927 0.736 0.625 0.749 $\tau$-norm 0.899 0.779 0.739 0.815 0.916 0.734 0.652
0.756 Eqvl2-FL 0.899 0.762 0.669 0.789 0.932 0.734 0.595 0.738 LDAM-FL 0.887
0.759 0.654 0.779 0.925 0.722 0.555 0.716 Focal-FL 0.879 0.747 0.610 0.759
0.929 0.710 0.573 0.721 Fed-GBA 0.883 0.791 0.773 0.823 0.921 0.714 0.695
0.768

Table 9: Test accuracies of various methods on IID partitioned CIFAR-10-LT
with diverse imbalance factors. Both the overall test accuracies and category-
wise accuracy are included.

##  Appendix E Further Analysis

###  E.1 Static and dynamic $\Delta_{j}(t)$

In the main paper, we show that the value of static $\Delta_{j}(t)$ does not
matter, as the model will reach to similar performance with a difference
$\Delta_{j}(t)$. Here we raise a question: whether is there an optimal dynamic
$\Delta_{j}(t)$ in long-tailed learning? One of the conjectures is that it
does contain a natural optimal trend of $\Delta_{j}(t)$, termed as “best re-
weighting”, but it has not been understood yet.

However, in the Fed-LT setting, $\Delta_{j}(t)$ trends of best re-weighting
are highly related to numerous systematic parameters, e.g., the degree of
long-tail and heterogeneity, class numbers, sampling strategy, client numbers,
and size of the dataset. It is unrealistic to design or compute the best trend
for each client with diverse data distribution. Even though it is possible to
obtain the potential best re-weighting strategy, it is still hard to directly
deploy the centralized long-tail re-weighting method to local training due to
well-known discrepancies or inconsistencies across local and global sides in
Fed-LT. Such challenges motivate us to adopt static $\Delta_{j}(t)$ for all
clients in our proposed Fed-GraB.

###  E.2 Plug-and-play class-wise performance boosting with SGB

In Fed-GraB, SGB is mounted on all classes and performs with the combination
of the prior vector instead of mounting on specifically designed tailed
classes without global prior. Here, we show that SGB can be flexibly mounted
on a customized tail class (i.e., plug-and-play) without global prior for
performance boosting. As shown in Fig. 12, though Fed-GraB could effectively
improve the accuracy on all middle and tailed classes and maintain the
performance of head classes. The scheme “Mounted on Class 6” could
significantly improve the performance of the tail Class 6.

Figure 12:  Plug-and-play performance boosting with SGB. Figure 13: Tail
estimation performance with DPA when tail part shifts by a different
threshold, the value of which ranges from $15$ to $90$.

###  E.3 Full-scale accuracy of DPA

To further demonstrate the effectiveness of DPA, we shift the threshold of
tail categorization to a full-scale level. We set
$\text{IF}_{\text{F}}=100/50/10$ and $\alpha=0.5$. After training 80 rounds
with FedAvg. The weights of global classifiers are used to generate the prior
vector via DPA. Then we sort the distribution from large to small as a long-
tailed curve. The thresholds are selected in $15\%-90\%$, as illustrated in
Fig. 13. It shows that DPA can achieve good overall performance in all class
sections, and reach slightly higher estimation accuracy for extremely long-
tailed data.

###  E.4 The effects of mounting strategy

In the experiments, masking top classes from mounting SGB is necessary for
backbone training, because it could learn good representations. Hence, Fed-
GraB remains top class gradients unchanged for better feature extractor and
re-weights the tailed part for better classifiers.

To show the effectiveness of the used “mounting on the tail classes” strategy,
we compare mounting SGB on the tail (Fed-GraB-95) and the middle&tail (Fed-
GraB-75) in Tab. 10. Based on the pre-trained model by FedAvg, we further
train for 300 rounds. The results show that the two strategies have similar
performance and “mounting on the tail classes” enjoys lower computational
costs.

Setting | Method |  $\text{IF}_{\text{G}}$=50 |  $\text{IF}_{\text{G}}$=100
---|---|---|---
Many | Med | Few | All | Many | Med | Few | All
$\alpha$=1 | Fed-GraB-95 | 0.957 | 0.809 | 0.756 | 0.818 | 0.960 | 0.789 | 0.683 | 0.781
Fed-GraB-75 | 0.962 | 0.772 | 0.819 | 0.829 | 0.963 | 0.723 | 0.758 | 0.785
$\alpha$=0.5 | Fed-GraB-95 | 0.946 | 0.795 | 0.796 | 0.826 | 0.956 | 0.771 | 0.680 | 0.772
Fed-GraB-75 | 0.957 | 0.750 | 0.835 | 0.826 | 0.968 | 0.700 | 0.744 | 0.771

Table 10: Performance evaluation of different SGB mounting strategies

###  E.5 Tracking of all clients

SGB is an effective gradient balancer with generalization capability, which is
not limited by class types or local distributions. We visualize the trend of
$\Delta_{j}(t)$ for head, middle, and tail classes in Fig. 14. It shows that
SGB can work well with clients with diverse distributions. Note that debt
classifiers will be out of control caused by no positive gradients passing by,
and we propose a Class Activation strategy (in Sec. C) to address this issue.

Figure 14: Tracking visualization of 10 selected clients on CIFAR-100-LT, with
$\text{IF}_{\text{G}}=50$ and $\alpha=0.5$.

###  E.6 T-SNE visualization

We additionally visualize model outputs via t-SNE ( [62]); See Fig. 15. The
mapping points of the uniform category are relatively concentrated. Category
clusters are further apart so that it is well distinguished and there is less
internal noise (from other categories). It demonstrated that our model
achieves high discrimination.

|
---|---
(a) | (b)
Figure 15: T-SNE visualization on CIFAR-10-LT. Left: For FedAvg, the samples
from different classes are mixed, especially in the central part. Right: For
Fed-GraB, both intersection area and individual class area decrease.