# Personalized Federated Learning with Server-Side Information

Jaehun Song  Min-hwan Oh  Hyung-Sin Kim

###### Abstract

Personalized Federated Learning (FL) is an emerging research field in FL that
learns an easily adaptable global model in the presence of data heterogeneity
among clients. However, one of the main challenges for personalized FL is the
heavy reliance on clients’ computing resources to calculate higher-order
gradients since client data is segregated from the server to ensure privacy.
To resolve this, we focus on a problem setting where the server may possess
its own data independent of clients’ data – a prevalent problem setting in
various applications, yet relatively unexplored in existing literature.
Specifically, we propose FedSIM, a new method for personalized FL that
actively utilizes such server data to improve meta-gradient calculation in the
server for increased personalization performance. Experimentally, we
demonstrate through various benchmarks and ablations that FedSIM is superior
to existing methods in terms of accuracy, more computationally efficient by
calculating the full meta-gradients in the server, and converges up to 34.2%
faster.

Machine Learning, Federated Learning, Meta Learning, Personalization

Graduate School of Data Science, Seoul National University, Seoul, South Korea
{steve2972}, {minoh}, {hyungkim} @snu.ac.kr

##  1 Introduction

Federated Learning (FL) has drawn significant attention from the research
community in recent years due to its potential for privacy-centric machine
learning in distributed learning environments. However, one challenge of FL
that remains prevalent today is diverse, non-i.i.d. data distributions among
clients, which limits a single global model from delivering optimal
performance on each client’s task.

One of the recent research directions that address this issue is personalized
federated learning, a personalized variant of FL based on techniques used in
optimization-based meta-learning such as in the Model-Agnostic Meta-Learning
(MAML) framework (Finn et al., 2017). The goal of personalized FL is to create
an adaptable global model parametrized by $\theta$ in a federated environment
such that $\theta$ can easily be fine-tuned to each client’s individual task
with a small number of gradient steps. To achieve this goal, personalized FL
optimizes

$\min_{\theta\in\mathbb{R}^{d}}F(\theta):=\frac{1}{n}\sum_{i=1}^{n}f_{i}(\theta-\eta\nabla
f_{i}(\theta;\mathcal{D}_{i}))$ (1)

where $f_{i}:\mathbb{R}^{d}\rightarrow\mathbb{R}$ is the loss corresponding to
client $i$, $\eta\geq 0$ is the step-size, and $\mathcal{D}_{i}$ is a batch of
data for client $i$ which follows the distribution $p_{i}$ where each client’s
data is assumed to be heterogeneous.

However, a major challenge of applying meta-learning to FL is the necessity to
calculate second-degree meta-gradients (Fallah et al., 2020b; Chen et al.,
2019). These gradients require not only additional data sampled from each
client’s data distribution $p_{i}$, but also additional computation for
clients to obtain the required Hessian matrix. Hence, computing the Hessian
poses a significant bottleneck in resource-constrained environments for
personalized FL.

To address this challenge, we aim to efficiently utilize computational
resources available in the entire system, acknowledging the disparity of the
computational overhead between clients and the central server in the existing
methods. Previous methods in personalized FL postulate that the central server
needs only to aggregate and average the optimized client weights to update the
global model. Here, the resourceful server is idle for most of the training
process while the resource-constrained clients are busy optimizing their local
models.

In this paper, we consider a variant of the personalized FL problem where the
server contains its own data. We denote this problem setting as Personalized
FL with Server Data. Server data is defined as data used to create and test a
model in the server before initiating the FL process and can be available in
various application domains. For example, hospitals and healthcare providers
may first test the validity of models using their own records before
implementing patient-wise predictions based on more privacy-sensitive
individual records. In predictive text, an initial predictive model can be
trained at the server with common phrases or words before implementing large-
scale FL for each client’s mobile device. In addition, an autonomous driving
company gathers its own data in various road conditions to train a model, but
can utilize FL to improve the model for each driver. However, most FL methods
use such server data only for creating an initial model and disregard it
during the FL process.

To this end, we propose a new method to estimate the computationally heavy
meta-gradients in the server by using server data. We propose that server data
can actively be utilized during the federated training process to augment
model performance, as described in Figure 1. We summarize our main
contributions as follows.

  * •

We propose a novel method FedSIM for Personalized Federated Learning with
Server Data. To our knowledge, FedSIM is the first personalized FL method that
efficiently utilizes the server’s computational and information resources to
compute the estimates of full meta-gradients with no additional client
computation compared to conventional FL.

  * •

The key components of our proposed method include (i) a custom loss with
$L_{2}$ regularization for local optimization, (ii) the approximation of
first-order meta gradients for each client by using the differences between
personalized model parameters and global model parameters, (iii) the
approximation of second-order meta gradients for each client using server data
without explicitly computing Hessian matrices.

  * •

The empirical evaluations demonstrate that FedSIM effectively improves model
performance even when the server has a relatively small amount of data
compared to the entire dataset ($\leq$5%), or when the distribution of server
data weakly represents that of non-i.i.d. data for each client.

  * •

We show that FedSIM outperforms existing methods in personalized FL. In
standardized FL benchmarks proposed in (He et al., 2020), FedSIM is up to
2.57% more accurate and requires 34.2% less communication rounds for
convergence.

Figure 1: Overall architecture of FedSIM. Clients focus solely on local
optimization while the server calculates meta-gradients for all clients using
server data.

##  2 Related Work

###  2.1 Federated Learning

Federated learning has rapidly evolved in various aspects (Li et al., 2020a),
with both empirical analyses (Bonawitz et al., 2019) and theoretical
guarantees (Hanzely et al., 2020) showing that FL models exhibit similar
performance to models trained in centralized data centers even when data does
not leave clients. In particular, there have been several works on the various
aspects of FL, including methods of reducing communication costs through
quantization (Amiri et al., 2020; Sun et al., 2019) or adaptive gradient
upload rounds (Wang et al., 2019; Amiri et al., 2020; Ivkin et al., 2019), and
convergence analyses with well-defined lower bounds (Hanzely et al., 2020;
Pathak & Wainwright, 2020; Wang et al., 2019).

A fundamental problem of FL is accuracy degradation due to training the model
with non-i.i.d. data across clients (Zhao et al., 2018). This problem is
significant because heterogeneous data distributions are common in practice
(Bonawitz et al., 2019) and thus investigated in a number of studies
(Haddadpour & Mahdavi, 2019; Khaled et al., 2020; Li et al., 2020c) with
solutions including normalized federated updates (Wang et al., 2020) and
computing stochastic gradients in minibatches (Woodworth et al., 2020).

###  2.2 Personalized Federated Learning

Personalized Federated Learning is a personalized variant of federated
learning that aims to improve model performance in non-i.i.d. data settings.
Examples include using Moreau envelopes (T. Dinh et al., 2020), model
interpolation (Mansour et al., 2020) and transfer learning-based
personalization (Arivazhagan et al., 2019; Chen et al., 2021).

In particular, FedMeta (Chen et al., 2019) and Per-FedAvg (Fallah et al.,
2020b) consider building upon the Model-Agnostic Meta-Learning (MAML)
formulation (Finn et al., 2017), and study the empirical and theoretical
success of the framework in a federated environment. However, these approaches
require the resource-constrained clients to locally execute full Hessian
calculations, thereby significantly increasing client-side computation and
memory overhead. A number of other works aim to decrease this computational
bottleneck by disregarding second-order calculations (Jiang et al., 2019),
inspired by first-order gradient-based meta learning approaches as in (Nichol
et al., 2018), while sacrificing model performance.

In contrast, FedSIM aims to calculate heavy meta gradients at the server using
server data to mitigate accuracy degradation due to disregarding the Hessians
without additional computational burden on clients.

##  3 Federated Learning with Server Information Meta-Learning (FedSIM)

###  3.1 Problem: Personalized FL with Server Data

In conventional FL, there are $n$ clients in a federated environment that
tries to find a global model $\theta$ by optimizing the following problem:

$\min_{\theta\in\mathbb{R}^{d}}f(\theta):=\frac{1}{n}\sum_{i=1}^{n}f_{i}(\theta)$
(2)

where $f_{i}:\mathbb{R}^{d}\rightarrow\mathbb{R}$ ($i=1,...,n$) denotes the
expected loss over the data distribution of client $i$ such that

$f_{i}(\theta)=\mathbb{E}_{\mathcal{D}_{i}\sim
p_{i}}[\ell_{i}(\theta;\mathcal{D}_{i})]$ (3)

where $\mathcal{D}_{i}$ is a random data sample drawn from client $i$’s data
distribution $p_{i}$ and $\ell_{i}(\theta;\mathcal{D}_{i})$ is the loss
corresponding with this data sample w.r.t. a global model parameter $\theta$.

In contrast to Eq. (2), we learn an adaptable global model $\theta$ in a
federated environment, by formulating and solving a bi-level (server- and
client-side) problem defined as

$\min_{\theta\in\mathbb{R}^{d}}F(\theta):=\frac{1}{n}\sum_{i=1}^{n}F_{i}(\theta)$
(4)
$F_{i}(\theta)=\min_{\phi_{i}\in\mathbb{R}^{d}}\tilde{f}_{i}(\phi_{i}):=f_{i}(\phi_{i})+\frac{\lambda}{2}||\phi_{i}-\theta||^{2}_{2}$
(5)

where $\phi_{i}$ denotes the personalized model of client $i$ and $\lambda$ is
a regularization parameter. Note that instead of the conventional loss in
Eq.(3), we define a different loss function $\tilde{f}_{i}(\phi_{i})$ that
includes an $L_{2}$ regularization term such that its gradient becomes

$\nabla_{\phi}\tilde{f}_{i}(\phi_{i})=\nabla_{\phi}f_{i}(\phi_{i})+\lambda(\phi_{i}-\theta)$
(6)

This custom loss is in accordance with ideas from (Li et al., 2020b; T. Dinh
et al., 2020) such that the personalized parameters $\phi_{i}$ are encouraged
to tend towards the global parameters $\theta$, which improves convergence of
the global model in non-i.i.d. data settings.

Lastly, we make a practical assumption that is not included in previous work
on personalized FL; the server has its own data with distribution $p_{s}$
independent of clients’ data. Here, we also assume that the proportion of
server data is small compared to the entire dataset.

###  3.2 FedSIM: A FL Framework for Server Utilization

Federated Learning with Server-Side Information Meta-Learning (FedSIM) is a
personalized FL framework that aims to (i) ensure client data privacy with
(ii) minimal additional computation/communication overhead in clients compared
to FedAvg in order to (iii) produce high-quality meta gradients. In this
section, we present the FedSIM algorithm to solve for Eq.(4) in the context of
Eq.(5).

In vanilla FL (e.g., FedAvg in McMahan et al. 2017), a centralized server
computes a global model by averaging models from decentralized devices. At
each round $t$, the server samples a client subset $S_{t}$ of size $m$ to
optimize the global model $\theta_{t-1}$. Each client $i\in S_{t}$ updates
$\theta_{t-1}$ with its private data $\mathcal{D}_{i}\sim p_{i}$ using
gradient decent for $E$ epochs and uploads the optimized model $\phi_{i}$ back
to the server. Finally, the server updates the global model to $\theta_{t}$ by
averaging $\phi_{i}$ received from $S_{t}$. It is important to note that
$\mathcal{D}_{i}$ was never shared between the clients nor with the server.

The main contribution of this work comes from allocating the calculation of
$\nabla_{\theta}F_{i}(\theta)$ to optimize Eq.(4) between the clients and the
server such that the server can calculate meta-gradients for multiple tasks
without sharing data. To this end, FedSIM follows the same principles as FL,
but with additional computation for meta-gradients in the server to learn an
easily adaptable global model.

Algorithm 1 FedSIM: Client-Side

0: Step size $\alpha$, regularization strength $\lambda$, client data
distribution $p_{i}$

function ClientUpdate$(i,\theta)$: // Run on client $k$

$\phi_{i,0}\leftarrow\theta$

for each local epoch $e$ from $1$ to $E$ do

Sample a mini-batch $\mathcal{D}_{i}$ from distribution $p_{i}$

Calculate $\phi_{i,e}$ using $\mathcal{D}_{i}$ with Eq.(7)

end for

Return $\phi_{i,E}$ to server

end function

Client-side algorithm. The client’s main goal is to learn a personalized model
$\phi_{i}$ by calculating local gradient updates at each local epoch $e$ as

$\phi_{i,e}=\phi_{i,e-1}-\alpha\nabla_{\phi}\tilde{f}_{i}(\phi_{i,e-1})$ (7)

To calculate $\nabla_{\phi}\tilde{f}_{i}(\phi_{i,e-1})$ in practice, we use an
unbiased estimate $\nabla_{\phi}\ell_{i}(\phi_{i,e-1};\mathcal{D}_{i})$ by
sampling a mini-batch of data $\mathcal{D}_{i}$ from distribution $p_{i}$.
This process is illustrated in Algorithm 1.

Algorithm 2 FedSIM: Server-Side

0: Step size $\beta$, $\delta$, server data distribution $p_{s}$

Initialize $\theta_{0}$

for each round $t=1,2,...$ do

Sample a mini-batch $\mathcal{D}_{s}$ from distribution $p_{s}$

$S_{t}\leftarrow$ Random subset of $m$ clients ($1\leq m\leq n$)

for each client $i\in S_{t}$ in parallel do

$\phi_{i}\leftarrow$ ClientUpdate($i$, $\theta_{t-1}$) (Algorithm 1)

Calculate $v_{i}=\nabla_{\phi}f_{i}(\phi_{i})$ with Eq.(9)

Calculate $d_{i}=\nabla^{2}_{\phi}f_{i}(\phi_{i})v_{i}$ using
$\mathcal{D}_{s}$ with Eq.(10)

Calculate meta-gradient $\nabla_{\theta}F_{i}(\theta_{t-1})=v_{i}-\delta
d_{i}$

Update
$\tilde{\phi}_{i}\leftarrow\phi_{i}-\beta\nabla_{\theta}F_{i}(\theta_{t-1})$

end for

$\theta_{t}\leftarrow\frac{1}{m}\sum_{i\in S_{t}}\tilde{\phi}_{i}$

end for

Server-side algorithm. The server then attempts to optimize Eq.(4) for
multiple communication rounds in Algorithm 2. In each round $t$, the central
server (i) samples $m$ clients, (ii) calculates meta-gradients
$\nabla_{\theta}F_{i}(\theta_{t-1})$ for each of these clients using local
model $\phi_{i}$ and server data $\mathcal{D}_{s}$ and (iii) updates the
global model from $\theta_{t-1}$ to $\theta_{t}$ using these meta-gradients.

As shown in (Rajeswaran et al., 2019), the gradient of Eq.(5) w.r.t. $\theta$
with the local loss function $\tilde{f}_{i}(\phi_{i})$ can be written as

$\displaystyle\nabla_{\theta}F_{i}(\theta_{t-1})=\left(\textbf{I}+\frac{1}{\lambda}\nabla^{2}_{\phi}f_{i}(\phi_{i})\right)^{-1}\nabla_{\phi}f_{i}(\phi_{i})$
(8)

Note that $\nabla_{\theta}F_{i}(\theta_{t-1})$ is not dependent on the
original meta model $\theta_{t-1}$, while corresponding with the personalized
model $\phi_{i}$. This characteristic comes from the regularization term in
Eq. (5) (Rajeswaran et al., 2019). Since the meta-gradient
$\nabla_{\theta}F_{i}(\theta_{t-1})$ is decoupled from $\theta_{t-1}$, the
server approximates the meta-gradient without requiring a history of client
$i$’s local updates. This allows clients to utilize multi-step gradient decent
for local optimization.

We can see from Eq.(8) that the calculation of
$\nabla_{\theta}F_{i}(\theta_{t-1})$ requires two terms:

  1. 1.

A first-order gradient $v_{i}=\nabla_{\phi}f_{i}(\phi_{i})$

  2. 2.

A Hessian-vector product $d_{i}=\nabla^{2}_{\phi}f(\phi_{i})v_{i}$

Unlike previous meta-learning approaches to personalized FL, we propose to
calculate both $v_{i}$ and $d_{i}$ using the server, without requiring
additional information or computation from clients.

First-order meta-gradient. As in Per-FedAvg, the first-order meta-gradient
$v_{i}$ ideally requires a client-specific query dataset $\mathcal{D}_{i}^{q}$
to calculate an unbiased estimate
$\nabla_{\phi}\tilde{f}_{i}(\phi_{i};\mathcal{D}_{i}^{q})$. However, in
FedSIM, since the server does not have the required client data, we instead
approximate $v_{i}$ by using the weight difference between a personalized
model $\phi_{i}$ and global model $\theta$ such that

$v_{i}=\nabla_{\phi}f_{i}(\phi_{i})\approx\theta-\phi_{i}$ (9)

The intuition behind this method comes from the fact that the derivative of
$\nabla_{\phi}\tilde{f}_{i}(\phi_{i})$ in Eq.(6) at a stationary point
$\phi_{i}$ becomes sufficiently small.

A possible alternative to calculate $v_{i}$ at the server is to sample a query
dataset $\mathcal{D}^{q}_{s}$ from server data distribution $p_{s}$ and
calculate $\nabla_{\phi}\ell_{i}(\phi_{i};\mathcal{D}^{q}_{s})$. A potential
drawback in this approach is that $\mathcal{D}^{q}_{s}$ does not come from
data distribution of client $i$. Our ablation study in Section 4 shows that
the weight difference approximation is superior to direct calculation using
server data.

Second-order meta-gradient. To calculate $d_{i}$, instead of separately
computing the Hessian $\nabla^{2}_{\phi}f_{i}(\phi_{i})$, we approximate the
entire Hessian-vector product $d_{i}$ by using Hessian-free estimation (Fallah
et al., 2020a) as follows:

$\displaystyle d_{i}$ $\displaystyle=$
$\displaystyle\nabla^{2}_{\phi}f_{i}(\phi_{i})v_{i}$ (10)
$\displaystyle\approx$ $\displaystyle\frac{\nabla_{\phi}f_{i}(\phi_{i}+\delta
v_{i})-\nabla_{\phi}f_{i}(\phi_{i}-\delta v_{i})}{2\delta}$

This approximation produces an error of at most $\rho\delta\|v_{i}\|^{2}$,
where $\rho$ is the parameter for Lipschitz continuity of the Hessian of $f$
(Fallah et al., 2020a).

Ideally, calculating unbiased estimates for the two first-order gradients
$\nabla_{\phi}f_{i}(\phi_{i}+\delta v_{i})$ and
$\nabla_{\phi}f_{i}(\phi_{i}-\delta v_{i})$ in Eq.(10) requires additional
client-specific data. We take an alternative approach since the server does
not have client data. The server samples $\mathcal{D}_{s}$ from its own data
distribution $p_{s}$ and calculates $\nabla_{\phi}\ell_{i}(\phi_{i}+\delta
v_{i};\mathcal{D}_{s})$ and $\nabla_{\phi}\ell_{i}(\phi_{i}-\delta
v_{i};\mathcal{D}_{s})$.

Note that we reuse the same dataset $\mathcal{D}_{s}$ to calculate $d_{i}$ for
all clients in $S_{t}$. Given that the server data distribution $p_{s}$ is
likely to be different from each client’s data distribution, the quality of
$d_{i}$ calculated using $\mathcal{D}_{s}$ may not be ideal. Nevertheless, we
hypothesized that using the non-ideal second-order terms would improve
performance over disregarding the second-order terms altogether. The
effectiveness of this approximation will be empirically evaluated in Sections
4 and 5.

Key differences. Compared to Per-FedAvg (Fallah et al., 2020b), a recent meta-
learning method for personalized ML, FedSIM does not calculate meta-gradients,
which is a computationally expensive operation, at resource-constrained
clients but at the server. Moreover, FedSIM is not restricted to one-step
gradient update when calculating $\phi_{i}$ at clients but allows multi-step
updates. pFedMe (T. Dinh et al., 2020) and FedProx (Li et al., 2020b) are
similar to FedSIM in that the client-side problem in Eq.(5) includes a
regularization term and each client utilizes multi-step gradient decent to
obtain its optimized model $\phi_{i}$. On the other hand, FedSIM enables meta-
learning without more computation on clients. Most importantly, FedSIM
actively utilizes the server to both aggregate personalized models and
calculate computationally heavy meta-gradients by utilizing server data.

Table 1: Non-i.i.d. datasets and model architectures for federated learning
benchmark (He et al., 2020).

Datasets | # of training | Non-i.i.d. | # of partitions | # of partitions | Baseline
---|---|---|---|---|---
samples | partition method | for clients | reserved server data | model architecture
Federated EMNIST | 671585 | realistic | 3230 | 0$-$170 | CNN (2 Conv + 2 FC)
CIFAR-100 | 50000 | Pachinko | 475 | 0$-$25 | ResNet-18 + group normalization
Shakespeare | 16068 | realistic | 680 | 0$-$35 | RNN (2 LSTM + 1 FC)
StackOverflow | 135818730 | realistic | 325354 | 0$-$17123 | RNN (1 LSTM + 2 FC)

Figure 2: Effect of the proportion of server data on method performance when
$E=5$.

###  3.3 Key Components for Fed-SIM

The key components for the FedSIM framework can be summarized as follows:

  * •

Custom loss for local optimization: Each client adds an $L_{2}$ regularization
term to its loss function as in Eq.(5) when optimizing a global model locally.
This decouples meta gradient calculation (at the server) from local
optimization history (at the clients).

  * •

First-order meta gradient calculation using weight differences: Despite the
existence of server data, the server calculates the first-order gradient
$v_{i}$ using (client-specific) weight differences as in Eq.(9) instead of
using the server data.

  * •

Second-order meta gradient calculation using server data: The server
calculates second-order gradient $d_{i}$ in a Hessian-Free way as in Eq.(10).
The approximation requires the two terms calculated using server data
$\mathcal{D}_{s}\sim p_{s}$ as $\nabla_{\phi}f_{i}(\phi_{i}+\delta
v_{i};\mathcal{D}_{s})$ and $\nabla_{\phi}f_{i}(\phi_{i}-\delta
v_{i};\mathcal{D}_{s})$.

With these components, FedSIM ensures that data remains on the client while
also ensuring that the calculation and communication done on the client is no
more intensive than that done during standard federated learning.

##  4 Experiments

The goal of our experiments is to evaluate (i) the performance of FedSIM
compared with existing methods on personalized FL with non-i.i.d. client data,
(ii) the convergence and computational overhead of FedSIM, and (iii) the
effectiveness of the three key components for FedSIM. All our experiments were
simulated using a server comprising four NVIDIA RTX 3900 GPUs and two Intel
Xeon Silver CPUs. To our knowledge, this section also serves as the most
comprehensive empirical study on personalized FL.

###  4.1 Experimental Design

Benchmarks. We compare FedSIM with other personalized FL methods based on
optimization-based meta-learning, FedMeta (Chen et al., 2019), Fed-Reptile
(Jiang et al., 2019), Per-FedAvg (FO)111Per-FedAvg (FO) is the first-order
approximation of Per-FedAvg, which makes clients compute first-order meta
gradients but disregard second-order terms. The full version of Per-FedAvg is
the same as FedMeta. (Fallah et al., 2020b), and pFedMe (T. Dinh et al.,
2020), and also regular FL methods FedAvg (McMahan et al., 2017) and FedProx
(Li et al., 2020b). Note that since FL is a newly growing research field,
existing work have used their own benchmarks to evaluate their respective
methodologies, with their own methods of splitting data in a non-i.i.d.
manner, which made it difficult to provide a fair comparison in performance.

To mitigate this problem, the authors of FedML (He et al., 2020) opened a
research library including benchmarks for federated learning. Thus, we use
four non-i.i.d. datasets, Federated EMNIST (Caldas et al., 2019), CIFAR-100
(Krizhevsky, 2009), Shakespeare (McMahan et al., 2017), and StackOverflow
(Authors, 2021), and train a standardized neural network for each dataset with
experiments constructed as suggested in (He et al., 2020). While three of
these datasets are naturally partitioned with a non-i.i.d. distribution,
CIFAR-100 is partitioned using Pachinko Allocation Method as in (Reddi et al.,
2020). The exact specifications are summarized in Table 1.

Figure 3: Effect of the number of local epochs $E$ for fine-tuning $\phi_{i}$
when server data proportion=5%.

Server Data Simulation. For each dataset, we randomly sample 5% of the
non-i.i.d. data partitions and reserve them as server data, while using the
remaining 95% partitions as client datasets. It is important to note that
while all the methods use server data for training an initial model, only
FedSIM uses server data during the actual FL process. Given that FedSIM takes
advantage of server data, we also experimented with different amounts of
server data.

Furthermore, when running Per-FedAvg (FO) and FedMeta, 80% and 20% of each
client’s training data are allocated as the client’s support and query
datasets respectively for local calculation of meta gradients.

Training Process. Training is carried out with $m=10$ as in (Li et al., 2020b;
Reddi et al., 2020), such that $m$ clients are randomly sampled in each round
to perform local optimization. Test accuracy is evaluated every round by
sampling $m$ clients, deploying the current global model, fine-tuning
(personalization) on each client’s training data, and finally averaging the
validation accuracy of all clients. Note that since FedProx and FedAvg do not
provide a personalization step, we add an update step to simulate
personalization of the global model.

Table 2: Best performance of each method on each dataset.

Methodologies | Fed. EMNIST | CIFAR-100 | Shakespeare | StackOverflow
---|---|---|---|---
FedProx \+ update | 88.03 | 51.82 | 57.55 | 27.41
FedAvg + update | 83.66 | 41.49 | 54.28 | 25.21
pFedMe | 89.26 | 52.31 | 59.02 | 27.91
Per-FedAvg (FO) | 86.17 | 50.99 | 58.34 | 27.95
FedMeta | 89.77 | 46.29 | 51.46 | 26.72
Fed-Reptile | 85.84 | 51.96 | 55.85 | 27.29
FedSIM (ours) | 91.51 $\pm$ 0.273 | 53.09 $\pm$ 0.104 | 60.81 $\pm$ 0.329 | 28.17 $\pm$ 0.171

Figure 4: Test accuracy as communication round increases when $E=10$ and
server data proportion is 5%.

###  4.2 Method Performance

Effects of Server Data Proportion. Figure 2 shows the average test accuracy of
various personalized FL methods with varying amounts of server data. Note that
while the amount of server data varies from 0 to 5%, 95% of the entire dataset
are always allocated as clients. The results show that more server data
results in better accuracy in all methods, implying that training an initial
model using server data improves performance. FedAvg shows the worst
performance since it does not train an adaptable (personalizable) model.
Although the performance of other five conventional methods vary by data
settings, FedSIM always provides the best accuracy once server data is given.

In particular, with 5% server data, our method’s performance exceeds all other
values in every dataset. As shown in Table 2, when comparing the best values
in each dataset, FedSIM provides 0.22$-$2.57% higher accuracy than the next
best methods. This verifies that FedSIM’s meta-gradient computation is an
effective way for using server data during the FL process even when server
data is not representative of the entire dataset.

Note that server data is not ideal since conventional MAML requires task
(client)-specific datasets for meta gradients. However, our results suggest
that if the client datasets are not given to the server due to privacy
concerns, calculation of second-order meta gradients using the server data can
be a good alternative rather than giving up the second-order terms as in Per-
FedAvg (FO), pFedMe, and Fed-Reptile. When there is no server data, FedSIM
cannot calculate Hessian estimates, essentially becoming the same as pFedMe.
In this setting, however, FedSIM still outperforms both Fed-Reptile and
FedProx, showing that the implementation of both a custom loss and first-
gradient estimates results in more accurate meta-gradients by preventing local
model divergence.

Furthermore, FedSIM shows that utilizing the server to directly calculate meta
gradients is more effective than simply averaging locally trained meta models
as in FedMeta. Note that FedMeta enables each client to calculate full meta
gradients including second-order terms on its client-specific dataset when
optimizing its local model, which requires heavy computation on clients but
turns out to be not effective for improving the global model.

Effects of Local Epochs. Figure 3 shows test accuracy of the same methods with
5% server data and varying local epochs $E$. While all the methods show better
accuracy as $E$ increases, FedSIM experiences remarkable improvement when $E$
increases from 1 to 5 and regularly outperforms all the other methods when
$E\geq 5$. In each dataset, the best accuracy value is given by FedSIM with
$E=20$, showing an 1.09$-$2.57% increase in accuracy compared to the second
highest values.

###  4.3 Resource Efficiency

Figure 5: Effect of epochs on average per-client, per-round computation time
(seconds).

Next, we evaluate the resource efficiency of FedSIM in terms of local
computation and communication overhead. Figure 4 plots test accuracy as
communication round ($t$) increases. While FedSIM achieves the highest
accuracy in all cases, it achieves the next best accuracy in 34.2%, 11.38%,
19.44%, 20.07% fewer communication rounds for each respective dataset due to
the use of more accurate meta gradients for model updates. Given that all the
methods require the same communication overhead in each round (i.e.,
dissemination of $\theta$ and aggregation of $\phi_{i}$), fewer rounds entail
less communication overhead.

Figure 5 shows the average client computation time for local optimization in
each round. The computation time of FedMeta (or Per-FedAvg) quickly increases
with local epochs due to local calculation of second-order meta gradients.
Per-FedAvg (FO) ignores the second-order terms but still calculates first-
order meta gradients locally, resulting in the second-longest computation
time. On the other hand, FedSIM shows a modest increase in client computation
time as local epoch increases, similar to FedAvg that does not provide
personalization, since meta gradients are calculated at the server. Overall,
FedSIM not only trains a more accurate model but also does so resource-
efficiently.

###  4.4 Ablation Studies

Table 3: % accuracy of FedSIM variants with $E=5$ and 5% server data.

Methodologies | Loss | FO meta | SO meta | Federated | CIFAR-100 | Shakespeare | StackOverflow
---|---|---|---|---|---|---|---
function |  gradients | gradients | EMNIST
FedSIM-var1 | basic | weight diff | server | 90.80 | 49.73 | 55.68 | 26.81
FedSIM-var2 | custom | server data | server | 77.78 | 48.45 | 53.49 | 25.76
FedSIM-var3 | custom | weight diff | x | 85.86 | 51.01 | 57.26 | 27.12
FedSIM | custom | weight diff | server | 91.51 | 53.09 | 60.81 | 28.17

Given that the distribution of server data is dissimilar to that of each
client’s data, using server data without caution may end up with performance
degradation. To this end, we evaluate if each key component of FedSIM actually
contributes to its performance, namely the (i) loss function, (ii) first-order
(FO) meta gradient calculation, and (iii) second-order (SO) meta gradient
calculation. We made three variants of FedSIM, FedSIM-var1 that uses basic
loss function without $L_{2}$ regularization, FedSIM-var2 that calculates FO
meta gradients using server data instead of weight difference (i.e.,
$\nabla_{\phi}\tilde{f}_{i}(\phi_{i};\mathcal{D}^{q}_{s})$ where
$\mathcal{D}^{q}_{s}\sim p_{s}$), and FedSIM-var3 that disregards SO meta
gradients.

Table 3 shows the performance of these variants. Comparison with FedSIM-var3
verifies that although calculating SO meta gradients using client-independent
server data is not theoretically ideal, using the SO terms still results in
significantly better performance than relying only on FO meta gradients. The
FedSIM-var2 case shows, however, that the non-ideal server data causes severe
performance degradation when used for FO meta gradient calculation; using
(client-specific) weight differences is a better choice in case of calculating
FO meta gradients. In addition, FedSIM-var1 proves that using a custom loss to
decouple local optimization history from meta gradients and calculating meta
gradients based on $\phi_{i}$ (locally optimized model) rather than $\theta$
(previous meta model) result in more useful meta gradients. Overall, the
results verify that each of the key components of FedSIM highly impacts model
accuracy.

##  5 Data Dissimilarity Analysis

We can see in practical FL scenarios that although clients may have non-i.i.d.
data, the data distributions of clients are not entirely unrelated. Thus,
prior work such as FedProx (Li et al., 2020b), Per-FedAvg (Fallah et al.,
2020b), and pFedMe (T. Dinh et al., 2020), perform convergence analyses on the
global model by assuming that both data distributions and local gradients have
bounded dissimilarity among clients. FedSIM makes a similar assumption that
both server data distributions and meta gradients calculated using server data
have bounded dissimilarity. Thus, in this section, we conduct experiments to
evaluate the effect of distributional deviation between server and client
data.

First, we investigate data dissimilarity between the server and clients in
non-i.i.d. data settings with varying amount of server data. Next, we
empirically observe that the variance of data dissimilarity between the
clients and the server is an appropriate measure of model performance in
FedSIM.

Distribution Comparison. To investigate data dissimilarity, we randomly sample
a small percentage of data from two image datasets, Federated-EMNIST and
CIFAR-100, as in Section 4.1 to simulate server data. The average image
distribution of the server data is then compared to each of the remaining
clients, using a Structural Similarity Index (SSIM) (Wang et al., 2004) to
compare the image data. Thus, each comparison produces $SSIM_{(i,j)}$ for
$i\in\mathcal{D}_{s}$ and $j\in\mathcal{D}$ where $\mathcal{D}$ is the set of
all clients in a dataset and $\mathcal{D}_{s}$ is the set of server data. This
process is repeated many times for each proportion of server data, resulting
in boxplots in Figure 6.

Figure 6: Effects of varying amount of server data on image similarity
distributions.

Figure 6 shows different trends of data similarity in the two datasets.
Regarding CIFAR-100, there is a noticeable increase in SSIM with more server
data. This is contrary to SSIM in EMNIST, which remain fairly consistent. We
hypothesize that this is due to the fact that EMNIST is a relatively simple
dataset, not only represented in grayscale but also consisting of handwritten
letters that hardly differ by client, which leads to fast saturation of data
similarity with only a small amount of server data. On the other hand,
CIFAR-100 provides far more diverse images which require more server data such
that data similarity can converge (albeit at a lower SSIM than EMNIST), which
is more representative of real-world images.

Despite the differences, in both datsets, the general trend of the variance of
the similarity metrics decrease with more server data. In addition, SSIM is
higher than 0.75 even when server data proportion is 1%, showing that server
and client data distributions are not entirely unrelated.

Figure 7: Change in model performance in relation to the change in structural
similarity of images.

Relationship with Performance. We then analyze the impact of average image
similarity on model performance, as seen in Figure 7. This figure shows the
relationship between FedSIM performance on the EMNIST and CIFAR-100 datasets
and the variance in image similarity (SSIM) between the server data and each
of the client data. Data points were collected by training the models various
times with different amounts of server data. Then, the server data used during
each learning process is used to calculate image similarity with the remaining
client data. Finally, we measure the variance of image similarity for each
data point, and plot it with respect to the distribution of model accuracy.

Here, the middle of each box represents the mean variance in image similarity
for each server data proportion, while the error bars represent the variance
in test accuracy. The graphs show that there is a negative correlation between
SSIM variance and model performance. More server data results in better
accuracy due to its correlation with SSIM variance. This implies that even
with the same amount of server data, model performance can depend on the
method by which the server dataset is constructed.

##  6 Discussion

In this paper, we investigate a practical problem setting of FL, personalized
federated learning with server data. We adapt the meta-learning process to
create FedSIM where meta-gradients are calculated using the server to improve
model performance and reduce client computational overhead. We show that
FedSIM solves the proposed FL problem by first performing local optimization
using a custom loss function with a regularization term, and then using server
data with these locally optimized models to calculate the required gradients.
We also provide a variety of numerical experiments and ablations to illustrate
the performances of our method compared with existing methods in personalized
FL. Finally, we present empirical analyses on the distribution of server data
and its impact on performance.

While we focus on personalized FL and meta learning, we believe that this work
opens up an interesting avenue in the FL regime that investigates how powerful
server and its data can contribute to federated learning process effectively.

## References

  * Amiri et al. (2020) Amiri, M. M., Gunduz, D., Kulkarni, S. R., and Poor, H. V.  Federated learning with quantized global model updates, 2020.
  * Arivazhagan et al. (2019) Arivazhagan, M. G., Aggarwal, V., Singh, A. K., and Choudhary, S.  Federated learning with personalization layers, 2019.
  * Authors (2021) Authors, T. T. F.  Tensorflow federated stack overflow dataset.  https://www.tensorflow.org/federated/api_docs/python/tff/simulation/datasets/stackoverflow/load_data, 2021\.
  * Bonawitz et al. (2019) Bonawitz, K., Eichner, H., Grieskamp, W., Huba, D., Ingerman, A., Ivanov, V., Kiddon, C., Konečný, J., Mazzocchi, S., McMahan, B., Van Overveldt, T., Petrou, D., Ramage, D., and Roselander, J.  Towards federated learning at scale: System design.  In _Proceedings of Machine Learning and Systems_ , volume 1, pp. 374–388, 2019.
  * Caldas et al. (2019) Caldas, S., Duddu, S. M. K., Wu, P., Li, T., Konečný, J., McMahan, H. B., Smith, V., and Talwalkar, A.  Leaf: A benchmark for federated settings, 2019.
  * Chen et al. (2019) Chen, F., Luo, M., Dong, Z., Li, Z., and He, X.  Federated meta-learning with fast convergence and efficient communication, 2019.
  * Chen et al. (2021) Chen, Y., Wang, J., Yu, C., Gao, W., and Qin, X.  Fedhealth: A federated transfer learning framework for wearable healthcare, 2021.
  * Cohen et al. (2017) Cohen, G., Afshar, S., Tapson, J., and van Schaik, A.  Emnist: an extension of mnist to handwritten letters, 2017.
  * Fallah et al. (2020a) Fallah, A., Mokhtari, A., and Ozdaglar, A.  On the convergence theory of gradient-based model-agnostic meta-learning algorithms.  In _Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics_ , pp. 1082–1092, 2020a.
  * Fallah et al. (2020b) Fallah, A., Mokhtari, A., and Ozdaglar, A.  Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach.  In _Advances in Neural Information Processing Systems_ , volume 33, 2020b.
  * Finn et al. (2017) Finn, C., Abbeel, P., and Levine, S.  Model-agnostic meta-learning for fast adaptation of deep networks.  In _International Conference on Machine Learning_ , pp. 1126–1135, 2017.
  * Haddadpour & Mahdavi (2019) Haddadpour, F. and Mahdavi, M.  On the convergence of local descent methods in federated learning.  _arXiv preprint arXiv:1910.14425_ , 2019.
  * Hanzely et al. (2020) Hanzely, F., Hanzely, S., Horváth, S., and Richtarik, P.  Lower bounds and optimal algorithms for personalized federated learning.  In _Advances in Neural Information Processing Systems_ , volume 33, pp. 2304–2315, 2020.
  * He et al. (2020) He, C., Li, S., So, J., Zhang, M., Wang, H., Wang, X., Vepakomma, P., Singh, A., Qiu, H., Shen, L., Zhao, P., Kang, Y., Liu, Y., Raskar, R., Yang, Q., Annavaram, M., and Avestimehr, S.  Fedml: A research library and benchmark for federated machine learning.  _arXiv preprint arXiv:2007.13518_ , 2020.
  * Ivkin et al. (2019) Ivkin, N., Rothchild, D., Ullah, E., braverman, V., Stoica, I., and Arora, R.  Communication-efficient distributed sgd with sketching.  In _Advances in Neural Information Processing Systems_ , volume 32, 2019.
  * Jiang et al. (2019) Jiang, Y., Konečný, J., Rush, K., and Kannan, S.  Improving federated learning personalization via model agnostic meta learning, 2019.
  * Khaled et al. (2020) Khaled, A., Mishchenko, K., and Richtárik, P.  Tighter theory for local sgd on identical and heterogeneous data.  In _International Conference on Artificial Intelligence and Statistics_ , pp. 4519–4529, 2020.
  * Krizhevsky (2009) Krizhevsky, A.  Learning multiple layers of features from tiny images, 2009.
  * Li et al. (2020a) Li, T., Sahu, A. K., Talwalkar, A., and Smith, V.  Federated learning: Challenges, methods, and future directions.  _IEEE Signal Processing Magazine_ , 37(3):50–60, 5 2020a.  ISSN 1558-0792.
  * Li et al. (2020b) Li, T., Sahu, A. K., Zaheer, M., Sanjabi, M., Talwalkar, A., and Smith, V.  Federated optimization in heterogeneous networks.  In _Proceedings of Machine Learning and Systems_ , volume 2, pp. 429–450, 2020b.
  * Li et al. (2020c) Li, X., Huang, K., Yang, W., Wang, S., and Zhang, Z.  On the convergence of fedavg on non-iid data.  _International Conference on Learning Representations_ , 2020c.
  * Mansour et al. (2020) Mansour, Y., Mohri, M., Ro, J., and Suresh, A. T.  Three approaches for personalization with applications to federated learning, 2020.
  * McMahan et al. (2017) McMahan, B., Moore, E., Ramage, D., Hampson, S., and y Arcas, B. A.  Communication-efficient learning of deep networks from decentralized data.  In _Artificial Intelligence and Statistics_ , pp. 1273–1282, 2017\.
  * Nichol et al. (2018) Nichol, A., Achiam, J., and Schulman, J.  On first-order meta-learning algorithms, 2018.
  * Pathak & Wainwright (2020) Pathak, R. and Wainwright, M. J.  Fedsplit: an algorithmic framework for fast federated optimization.  In _Advances in Neural Information Processing Systems_ , volume 33, pp. 7057–7066, 2020.
  * Rajeswaran et al. (2019) Rajeswaran, A., Finn, C., Kakade, S. M., and Levine, S.  Meta-learning with implicit gradients.  In _Advances in Neural Information Processing Systems_ , volume 32, 2019.
  * Reddi et al. (2020) Reddi, S., Charles, Z., Zaheer, M., Garrett, Z., Rush, K., Konečnỳ, J., Kumar, S., and McMahan, H. B.  Adaptive federated optimization.  _arXiv preprint arXiv:2003.00295_ , 2020.
  * Sun et al. (2019) Sun, J., Chen, T., Giannakis, G., and Yang, Z.  Communication-efficient distributed learning via lazily aggregated quantized gradients.  In _Advances in Neural Information Processing Systems_ , volume 32, 2019.
  * T. Dinh et al. (2020) T. Dinh, C., Tran, N., and Nguyen, J.  Personalized federated learning with moreau envelopes.  In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H. (eds.), _Advances in Neural Information Processing Systems_ , volume 33, pp. 21394–21405. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper/2020/file/f4f1f13c8289ac1b1ee0ff176b56fc60-Paper.pdf.
  * Wang et al. (2020) Wang, J., Liu, Q., Liang, H., Joshi, G., and Poor, H. V.  Tackling the objective inconsistency problem in heterogeneous federated optimization.  In _Advances in Neural Information Processing Systems_ , volume 33, pp. 7611–7623, 2020.
  * Wang et al. (2019) Wang, S., Tuor, T., Salonidis, T., Leung, K. K., Makaya, C., He, T., and Chan, K.  Adaptive federated learning in resource constrained edge computing systems.  _IEEE Journal on Selected Areas in Communications_ , 37(6):1205–1221, 2019.
  * Wang et al. (2004) Wang, Z., Bovik, A., Sheikh, H., and Simoncelli, E.  Image quality assessment: from error visibility to structural similarity.  _IEEE Transactions on Image Processing_ , 13(4):600–612, 2004.  doi: 10.1109/TIP.2003.819861.
  * Woodworth et al. (2020) Woodworth, B. E., Patel, K. K., and Srebro, N.  Minibatch vs local sgd for heterogeneous distributed learning.  In _Advances in Neural Information Processing Systems_ , volume 33, pp. 6281–6292, 2020.
  * Zhao et al. (2018) Zhao, Y., Li, M., Lai, L., Suda, N., Civin, D., and Chandra, V.  Federated learning with non-iid data.  _arXiv preprint arXiv:1806.00582_ , 2018.

##  Appendix A Qualitative Comparison with Other Methods

Table 4 compares various aspects of existing methods in personalized federated
learningincluding FedSIM. Compared to existing methods in personalized
federated learning, FedSIM is differentiated in that it (1) utilizes a custom
loss for local optimization and (2) calculates full meta-gradient at the
server without additional communication overhead: first-order meta gradient
using weight difference and second-order gradient using Hessian-free
approximation and proxy data.

Table 4: Qualitative Comparison of existing methods in personalized federated
learning

Methodologies | FedAvg | Per-FedAvg | FedMeta | Fed-Reptile | FedSIM
---|---|---|---|---|---
(McMahan et al., 2017) |  (FO) (Fallah et al., 2020b) | (Chen et al., 2019) | (Jiang et al., 2019) | (ours)
Local | Loss function | general | general | general | general | L2 regularization
(task-specific) | Where to | client
optimization | compute
| Required | N/A | entire history, | entire history, | final weights | final weights
| information | query datset | query dataset
First-order | Method | N/A | exact | exact | weight | weight
meta gradient | calculation | calculation | difference | difference
(outer loop) |  |  | approximation | approximation
| Where to | N/A | client | client | server | server
| compute
| Required | N/A | N/A | entire history, | N/A | final weights,
Second-order | information | query dataset | proxy data
meta gradient | Method | N/A | N/A | exact | N/A | Hessian-free
(outer loop) | calculation | approximation
| Where to | N/A | N/A | client | N/A | server
| compute
Where training data is stored | Mostly on clients, (optionally) small amount of proxy data on server

##  Appendix B Experiment Details

In this section, we provide additional details of the experimental set-up for
the experiments in Section 4. We used federated versions of vision datasets
EMNIST (Cohen et al., 2017) and CIFAR-100 (Krizhevsky, 2009), alongside
language modeling datasets Shakespeare (McMahan et al., 2017) and
StackOverflow (Authors, 2021).

We train our model such that in each communication round, 10 clients are
sampled, the model is trained using each respective methodology on each
client’s training dataset. At the end of each communication round, we sample
10 different individuals, where each client is first fine-tuned using standard
training with no custom loss, and tests on its testing dataset using its own
fine-tuned model. We take the average of the client’s test accuracy to
evaluate the model’s performance. Note that we use $\lambda=1$ and
$\delta=0.25$.

Thus, each communication round can be summarized as the following.

  1. 1.

Sampling phase: where a number of clients are chosen from the entire client
pool, each with their own unique data randomly sampled from the training
dataset

  2. 2.

Training phase: where the model is trained to quickly adapt to each unique
client

  3. 3.

Testing phase: where the model is tested on a new client with data from the
test dataset

Furthermore, a summary of the hyperparameters we used for each dataset is
given in Table 5. Note that we fix the batch size at a per-task level given
the large number of hyperparameters to tune and to avoid conflating variables.

Table 5: Summary of hyperparameters used for each task Hyperparameters | Federated EMNIST | CIFAR-100 | Shakespeare | StackOverflow
---|---|---|---|---
Client Optimizer | Adam | Adam | Adam | Adam
Client Scheduler | x | x | x | x
Client Learning Rate | 0.01 | 0.001 | 0.001 | 0.001
Server Optimizer | SGD | SGD | SGD | SGD
Server Scheduler | Linear Decay | Linear Decay | Linear Decay | Linear Decay
Server Learning Rate | 0.25 | 0.25 | 0.25 | 0.25
Batch Size | 20 | 20 | 4 | 16

###  B.1 EMNIST

EMNIST (Cohen et al., 2017) consists of images of digits and upper and lower
case English characters, with 62 total classes. The federated version of
EMNIST (Caldas et al., 2019) partitions the digits by their author. The
dataset has natural heterogeneity stemming from the writing style of each
person. We perform an character recognition task using this dataset, with a
full description of the model in Table 6.

Federated EMNIST is partitioned in a manner such that 3,400 individuals
constitute a separate client, with each client having an individual training
dataset and a testing dataset. Thus, a testing round for EMNIST consists of
sampling a user, training on the user’s handwriting style, and testing on the
individual testing dataset for that particular user.

Table 6: Federated EMNIST model architecture. Layer | Output Shape | # of Trainable Parameters | Activation | Hyperparameters
---|---|---|---|---
Input | (28,28,1) | 0 |  |
Conv2d | (26,26,32) | 320 |  | kernel size=3; strides=(1,1)
Conv2d | (24,24,64) | 18496 | ReLU | kernel size=3; strides=(1,1)
MaxPool2d | (12,12,64) | 0 |  |
Dropout | (12,12,64) | 0 |  | $p=0.25$
Flatten | 9216 | 0 |  |
Dense | 128 | 1179776 |  |
Dropout | 128 | 0 |  | $p=0.5$
Dense | 62 | 7998 | softmax |

###  B.2 CIFAR-100

CIFAR-100 consists of images with RGB channels of 32x32 pixels each. Each
pixel is represented by an unsigned int8. As is standard with CIFAR datasets,
we perform preprocessing on the training images. For training images, we
augment the data by performing a random horizontal flip. We then scale the
pixel values such that each pixel value lies between $[0,1]$. We train a
modified ResNet-18 model, where the batch normalization layers are replaced by
group normalization layers.

Our model trains on a federated version of CIFAR-100 as proposed in (Reddi et
al., 2020), where the authors apply a two step latent Dirichlet allocation
(LDA) process by first randomly partitioning the data to reflect the ”coarse”
and ”fine” labels structure of CIFAR-100 by using the Pachinko Allocation
Method (PAM), and finally creating a federated dataset using LDA with a
parameter of 0.1. Using this method, the authors of (Reddi et al., 2020)
create a training dataset consisting of 500 clients and a testing dataset
consisting of 100 clients.

Although we train our model using the 500 training clients, we needed to
slightly modify the testing dataset in order to allow fine-tuning of the model
when deployed. To do so, for each test client, we split the client dataset
into a fine-tuning dataset and a validation dataset consisting of 80 and 20%
of the data respectively. By doing so, when testing, we sample ten clients
from the test client space, optimize the models on the fine-tuning dataset for
each client, and evaluate the models using each client’s respective validation
datasets.

###  B.3 Shakespeare

Shakespeare is a language modeling dataset built from the collective works of
William Shakespeare and first used in (McMahan et al., 2017) as a federated
learning task. The dataset consists of 715 ”actors”, each with their own
distinct method of talking. Each client’s lines are partitioned into training
and test sets. The natural language processing task here is to perform next
character prediction. To do so, we use an RNN that takes a series of 80
characters as input, passes it through the model, and outputs a sequence of
characters formed by shifting the input sequence by one. This way, the last
character is the new character we are actually trying to predict.

The model architecture for the Shakespeare character prediction task is shown
in Table 7.

Table 7: Shakespeare model architecture. Layer | Output Shape | # of Trainable Parameters
---|---|---
Input | 80 | 0
Embedding | (80, 8) | 720
LSTM | (80, 256) | 271360
LSTM | (80, 256) | 575312
Dense | (80, 90) | 23130

###  B.4 StackOverflow

StackOverflow is a language modeling dataset consisting of questions and
answers from the site Stack Overflow. The dataset contains 342,477 unique
users which we use as clients. We perform next-word prediction on this
dataset.

Preprocessing For this task, we restrict the dataset to 10,000 most frequently
used words, restrict each client such that they have at most 1000 sentences in
their dataset. We then truncate or pad the data such that each sentence has 21
words (20 words are used as the input sequence, and the last word is the
predicted output). We then represent the sentence as a sequence of indices
corresponding to the 10,000 most frequently used words. Our RNN model embeds
these sequences into a learned 96-dimensional space. It then feeds the
embedded words into a single LSTM layer, followed by two densely connected
layers with a softmax activation at the end. The model architecture for the
StackOverflow next word prediction task is shown in Table 8.

Table 8: StackOverflow model architecture Layer | Output Shape | # of Trainable Parameters
---|---|---
Input | 20 | 0
Embedding | (20, 8) | 960000
LSTM | (20, 256) | 2055560
Dense | (20, 256) | 64416
Dense | (10000) | 970000


# Quantum tricriticality and universal scaling in a tricritical quantum Rabi
system

You-Qi Lu  Department of Physics, and Chongqing Key Laboratory for strongly
coupled Physics, Chongqing University, Chongqing 401330, China  Yu-Yu Zhang
<EMAIL_ADDRESS>Department of Physics, and Chongqing Key Laboratory for
strongly coupled Physics, Chongqing University, Chongqing 401330, China

###### Abstract

Quantum tricriticality, a unique form of high-order criticality, is expected
to exhibit fascinating features including unconventional critical exponents
and universal scaling laws. However, a quantum tricritical point (QTCP) is
much harder to access, and the corresponding phenomena at tricriticality have
rarely been investigated. In this study, we explore a tricritical quantum Rabi
model, which incorporates a nontrivial parameter for adjusting the coupling
ratio between a cavity and a three-level atom. The QTCP emerges at the
intersection of a first- and second-order superradiant phase transitions
according to Landau theory. By using finite-frequency scaling analyses for
quantum fluctuations and the mean photon number, universal critical exponents
differentiate the QTCP from the second-order critical point. We find that the
phase transition at the tricritical point goes beyond the conventional second-
order phase transition. Our work explores an interesting direction in the
generalization of the well-known Rabi model for the study of higher-order
critical points due to its high control and tunability.

††preprint: APS/123-QED

Introduction –Quantum phase transition (QPT) is a central issue in the study
of many-body quantum phenomena at zero temperature [1]. Characterizing
universal phase transition phenomena and identifying critical exponents are
essential in understanding phase transitions. Quantum critical points are
often observed as a divergence point of an order parameter in continuous phase
transitions by adjusting external physical parameters such as magnetic fields
[2, 3]. In contrast to conventional critical points, a quantum tricritical
point (QTCP) arises where a continuous phase transition changes into a
discontinuous one. QTCPs were originally found in He3-He4 mixtures in finite
temperature phase diagrams, which were characterized by the Landau theory of
phase transitions [4]. Tricriticality is challenging to access in real
materials, but it can be found, for example, in itinerant ferromagnets [5] and
metallic magnets [6, 7, 8, 9]. Several experimental and theoretical works
indicate unconventional quantum criticalities resulting from quantum
tricriticalities in many-body systems [9, 10, 11, 12, 13, 14] .

QPTs in light-matter interaction systems have been extensively studied in
recent years, leading to the discovery of exotic quantum phases in quantum
many-body systems [15, 16, 17]. One well-known quantum phenomenon is the
superradiant phase transition, which occurs when a collection of two-level
atoms undergoes spontaneous emission [18, 19, 20, 21, 22], This phenomenon has
been observed in Bose–Einstein condensate gas [23] and degenerate Fermi gas
experiments [24]. The Quantum Rabi model, consisting of a two-level system and
a bosonic field mode, also exhibits a superradiant phase transition in a
infinite-frequency ratio limit with analogy to a thermodynamic limit [25, 26,
27, 28, 29]. This has been achieved in quantum simulations [30, 31].
Significant efforts have been dedicated to exploring the existence of QPTs in
few-body systems in finite Jaynes-Cummings lattice systems [32], anisotropic
quantum Rabi and Rabi-star model [33, 34, 35], and quantum Rabi ring with an
artifical field [36, 37, 38]. The QPTs in a few-body system offer an avenue
for investigating nontrivial criticality and exotic phases due to high degree
of tunability.

In this study, we explore a generalization of the well-known Rabi model,
aiming to find profound high-order criticality. We introduce a tricritical
quantum Rabi model that incorporates a tunable parameter that gauges the ratio
between the coupling strengths of the cavity and a three-level atom. By using
Landau theory, we expand the ground-state energy in terms of an order
parameter, revealing superradiant phase transitions of first and second
orders. Notably, a QTCP emerges at the boundary between critical lines for
first- and second-order quantum phase transitions. At the QTCP, the scaling
exponents of quantum fluctuations and the mean photon number are different
from exponents at the second-order critical point. Our findings indicate that
the QTCP belongs to a distinct universality class with a unique universal
critical exponent, differing from that of the conventional quantum Rabi model.

Figure 1: (a) An setup of the tricritical Rabi system. A three-level atom is
coupled to a single-mode cavity. Three levels of the atom are coupled by
cavity-assisted Raman transitions, for which the atomic transition ratio
$\gamma$ is tuned from the side by two driving lasers in red and yellow lines.
(b) Average value of the atom energy $\langle h\rangle$ in the
$\gamma-\lambda$ plane for the phase transitions from the NP
($\lambda<\lambda_{c}$) to the second-order SR and first-order SR phases
($\lambda>\lambda_{c}$), respectively. The white solid line is a second-order
critical line while the blue dashed line is a first-order critical line,
respectively. The QTCP is marked by a red dot. In all our calculations, we set
$\omega=1$ as the units for frequency.

Tricritical quantum Rabi model –We consider a tricritical quantum Rabi system,
which describes a three-level atom coupled uniformly to a single-mode cavity.
The Hamiltonian of this system is a generalization of the well-known quantum
Rabi model and reads

$H_{R}=\omega a^{\dagger}a+g(a^{\dagger}+a)d+\Omega h,$ (1)

where $a(a^{\dagger})$ denotes the photon annihilation (creation) operator of
the single-model cavity with the frequency $\omega$. $g$ is the atom-cavity
coupling strength, $\Omega$ characterizes the atom energy splitting. The
dipole operator $d$ of the atom and the single-atom Hamiltonian $h$ are
defined as

$d=\left(\begin{array}[]{ccc}0&1&0\\\ 1&0&\gamma\\\
0&\gamma&0\end{array}\right),~{}~{}~{}~{}~{}h=\left(\begin{array}[]{ccc}1&0&0\\\
0&0&0\\\ 0&0&-1\end{array}\right).$ (2)

The dipole operator $d$ incorporates a nontrivial parameter $\gamma$ that
tunes the strength ratio of the atomic transitions between
$\left|1\right\rangle\leftrightarrow\left|0\right\rangle$ and
$\left|0\right\rangle\leftrightarrow\left|-1\right\rangle$, for which
$\left|\varepsilon_{i}\right\rangle$ ($\varepsilon_{i}=0,\pm 1$) is the
eigenstates of $h$ of the three-level atom. $\gamma$ plays a crucial influence
on the effective coupling strength between the cavity and the atom. For a
experimental realization in Fig. 1 (a), a three hyperfine levels could be the
ones on the F$=1$ ground state of $87$Rb. Three levels are coupled through the
cavity and laser fields, which can be realized by cavity-assisted Raman
transitions [39]. The three-level atom interacts with two coupling laser with
different frequencies, which control the atomic transitions ratio $\gamma$. In
the following we denote a scaled dimensionless coupling strength as
$\lambda=g/\sqrt{\Omega\omega}$.

For a weak atom-cavity coupling $\lambda$, the excitation tends to zero, which
corresponds to the normal phase (NP). As $\lambda$ increases to a critical
value $\lambda_{c}$, the photon population becomes macroscopic, and the system
enters a superradiant (SR) phase. The SR phase transitions occurs in the
infinite-frequency limit by denoting $\eta=\Omega/\omega\rightarrow\infty$,
which is analogous to the infinite limit in the quantum Rabi model [26, 27].

Superradiant phases and a tricritical point –In the superrdiant phases, the
excitation is proportional to $\eta$ due to macroscopic population. Then we
follow a mean-field appraoch by shifting the bosonic operator with respected
to their mean value, $a\rightarrow a+\beta$ with $\beta=\langle
a\rangle\propto\sqrt{\eta}$. The effective ground-state energy term of the
Hamiltonian in Eq.(1) is obtained as

$\displaystyle H_{0}/\Omega$ $\displaystyle=$ $\displaystyle
h+\frac{2\lambda}{\sqrt{\eta}}\beta d+\frac{1}{\eta}\beta^{2},$ (3)

where $h$ and $d$ are the single atom operators given in Eq. (2). The ground-
state energy is determined by a nonzero value of $\beta$ minimizing the energy
of $H_{0}$ term. Fig. 1 (b) displays the average value $\langle h\rangle$
dependent on $\gamma$ and $\lambda$, which is obtained by numerical
variational method. In the NP regime with the coupling strength
$\lambda<\lambda_{c}$, $\langle h\rangle$ equals to $-1$ corresponding to the
atom in the lowest state. As $\lambda$ exceeds the critical value
$\lambda_{c}$, $\langle h\rangle$ smoothly increases from $-1$, revealing a
second-order SR phase transition. On the other hand, when $\gamma$ is below a
critical value $\gamma_{\texttt{TCP}}$, $\langle h\rangle$ changes sharply
with a discontinuous increasing, manifesting a first-order SR phase. Moreover,
the critical lines of the first and second-order phase transitions intersect
at a QTCP ($\gamma_{\texttt{TCP}},\lambda_{\texttt{TCP}}$) in a red dot. We
analyze the emergence of the first and second-order phase transitions in the
following using the Landau theory approach.

Figure 2: (a-c) Scaled ground-state energy $E_{g}/\eta$ and the mean photon
number $N_{ph}$ as a function of the dimensionless coupling strength
$\lambda/\lambda_{c}$ for $\gamma=0.1$ (a) and (d),
$\gamma=\gamma_{\texttt{TCP}}=1/\sqrt{2}$ (b) and (e), and $\gamma=1.0$ (c)
and (f) for $\eta=20,100,500$. The first and second derivatives of the energy
with respected to $\lambda$, $dE_{g}/d\lambda$ and $d^{2}E_{SR}/d^{2}\lambda$,
are listed in the insets.

Since the ground-state energy term in Eq.(3) can be reduced to
$H_{0}/\Omega=h+\alpha d+\alpha^{2}/4\lambda^{2}$ with the rescaled order
parameter $\alpha=2\lambda\beta/\sqrt{\eta}$. In the infinite-frequency limit
$\eta\rightarrow\infty$, the ground-state energy of the effective Hamiltonian
(3) can be expanded as a Taylor series in terms of the tiny value of $\alpha$,
$E_{SR}=\sum_{k=0}^{\infty}c_{k}\alpha^{2k}$. The coefficients $c_{k}$ are
obtained using the perturbation theory by treating $h$ as the unperturbed
Hamiltonian and $\alpha d$ term in Eq. (3) as the perturbation. Considering
the Landau theory, we perform the sixth-order perturbation energy by keeping
the expansion up to order $\alpha^{6}$ as

$\frac{E_{SR}}{\Omega}=c_{1}\alpha^{2}+c_{2}\alpha^{4}+c_{3}\alpha^{6}-1,$ (4)

where the coefficients $c_{1}=1/(4\lambda^{2})-\gamma^{2}$,
$c_{2}=\gamma^{2}(\gamma^{2}-\frac{1}{2})$, and
$c_{3}=-\gamma^{2}(1-7\gamma^{2}+8\gamma^{4})/4$ are given in the Appendix.
According to the first derivatives of $E_{SR}$ with respective to $\alpha$,
$dE_{SR}/d\alpha=0$, we obtain minimum values

$\displaystyle\alpha_{\pm}=\pm\sqrt{(-c_{2}+\sqrt{c_{2}^{2}-3c_{1}c_{3}})/3c_{3}},$
(5)

and $\alpha=0$. Obviuosly, the ordinary second-order critical boundary is
obtained when $c_{1}=0$ and $c_{2}>0$. It results in the second-order critical
boundary

$\displaystyle\lambda_{c}=\frac{1}{2\gamma}.$ (6)

It fits well with the critical line of the second-order phase transition in
Fig. 1(b) in white solid line.

The QTCP, where marks the intersection of first- and second-order phase
transitions, is determined from $c_{1}=c_{2}=0$ and $c_{3}>0$. It yields the
tricritical point

$\gamma_{\texttt{TCP}}=1/\sqrt{2},\lambda_{\texttt{TCP}}=1/\sqrt{2}.$ (7)

The location of the QTCP is marked in a red dot in Fig. 1(b).

When $\gamma\geq\gamma_{\texttt{TCP}}$, the ground-state energy $E_{SR}$ has
two global minima at $\alpha_{\pm}$ with the coefficient $c_{1}<0$, signaling
the second-order phase transition. For $\gamma<\gamma_{\texttt{TCP}}$,
$E_{SR}$ has three local minima at $\alpha_{\pm}$ and $\alpha=0$. As the phase
transition is crossed, the global minimum switches from $\alpha=0$ to
$\alpha_{\pm}$. The discontinuous jump of the energy in the global minimum
location indicates a first-order phase transition in Appendix.

Figure 3: The variance of momentum $(\Delta P)^{2}$ as a function of the
dimensionless coupling strength $\lambda/\lambda_{c}$ for $\gamma=1$ (a) and
$\gamma=\gamma_{\texttt{TCP}}$(b) for finite frequency ratio
$\eta=100,200,500$. Finite-frequency scaling functions for $\gamma=1$ (c) and
$\gamma=\gamma_{\texttt{TCP}}$ (d) for large values $\eta=1000,2000,5000$.

To show the validity of the perturbation theory, we accurately calculate the
scaled ground-state energy $E_{g}/\eta$ and the scaled mean photon number
$N_{ph}=\langle a^{\dagger}a\rangle/\eta$ of the Hamiltonian (1) by numerical
exact diagonalization. In the NP ($\lambda/\lambda_{c}<1$), the excitation of
photons $N_{ph}$ tends to zero due to zero excitation, while it increases in
the superradiant phase ($\lambda/\lambda_{c}>1$). For the transition strength
ratio $\gamma=0.1<\gamma_{\texttt{TCP}}$ in Fig.2 (a) (d), both
$dE_{g}(\lambda)/d\lambda$ and $N_{ph}$ are discontinuous at the critical
coupling strength $\lambda=\lambda_{c}$, revealing the first-order nature of
the QPT. When $\gamma=1>\gamma_{\texttt{TCP}}$ in Fig.2 (c) (f), $N_{ph}$
becomes continuous, while $d^{2}E_{g}(\lambda)/d^{2}\lambda$ is discontinuous,
indicating a second-order phase transition. The first- and second-order SR
phase transitions are consistent with the analysis using the Landau theory. At
the QCTP, $\gamma_{\texttt{TCP}}=1/\sqrt{2}$, Fig.2 (b) (e) show smooth
behavior of both $E_{g}$ and $N_{ph}$. Moreover, the critical point approaches
to the critical value $\lambda_{c}$ in Eq. (6) when $\eta$ grows from $20$ to
$500$, demonstrating the finite-frequency effect.

Figure 4: Scaling of the average photon number $N_{ph}$ (a) and variance of momentum $(\Delta P)^{2}$ (b) as a function of $\eta$ in log-log scale at the $2$nd-order critical point with $\gamma=1.0>\gamma_{c}$ (red circles) and the QTCP with $\gamma=\gamma_{\texttt{TCP}}$ (blue triangle). Finite-size scaling of the average photon number around the $2$nd-order critical point (c) and around the QTCP (d), respectively.  Table 1: Various critical exponents $\beta_{Q}$, $\beta_{Q}/\nu$ and $\nu$ obtained using the finite-frequency scaling function for the variance of momentum $\Delta P$, and the average photon number $N_{ph}$ for the QTCP and $2$nd-order critical point, respectively. |  | Critical exponent
---
$\beta_{Q}$
| Finite-frequency scaling
---
exponent $\beta_{Q}/\nu$
Universal critical exponent $\nu$
$(\Delta P)^{2}$ |  $2^{nd}$ order critical point | 1/2 | 1/3 | 3/2
QTCP | 1/2 | 1/2 | 1
$N_{ph}$ |  $2^{nd}$ order critical point | 1 | 2/3 | 3/2
QTCP | 1/2 | 1/2 | 1

Universal scaling and critical exponents –To gain universal features of
different phase transitions, it is of great interest to explore the critical
exponents and universality classes of the QTCP and the second-order QCP in the
tricritical Rabi model. It is well known that different systems can exhibit
similar quantum criticality, giving rise to universality. Finite-size scaling
is a topic of major interest in QPT systems and has been firmly established
since the development of a general theory [42, 43, 1].

The finite-size scaling ansatz for a physical quantity $Q$ in the critical
region takes the following scaling law form

$Q(\eta,\lambda)=\eta^{-\beta_{Q}/\nu}F_{Q}(|\lambda-\lambda_{c}|\eta^{1/\nu}),$
(8)

where $\nu$ is a universal critical exponent but independent of the physical
quantity, $F_{Q}(x)$ is the scaling function of $Q$, and $\beta_{Q}$ is the
critical exponent for $Q$. The scaling form dependent on $\eta$ is similar to
finite-size scaling in the thermodynamics phase transitions, which is known as
finite-frequency scaling.

At the critical point $\lambda_{c}$, one obtains log-log relation as

$\texttt{ln}Q(\eta,\lambda_{c})=-\frac{\beta_{Q}}{\nu}\texttt{ln}\eta+\texttt{ln}F_{Q}(0),$
(9)

where $\texttt{ln}F_{Q}(0)$ is a constant. The critical exponent
$\beta_{Q}/\nu$ is obtained as the slope of the linear dependence. It yields
the finite-frequency scaling relation as
$Q(\eta,\lambda_{c})\propto\eta^{-\beta_{Q}/\nu}$.

We consider the observable $Q$ as the variance $(\Delta P)^{2}=\langle
P^{2}\rangle-\langle P\rangle^{2}$ of the momentum quadrature
$P=i(a^{\dagger}-a)$, which is account for the quantum fluctuations. Fig. 3
(a)(b) shows the dependence of the $\Delta P$ on the coupling strength
$\lambda$. As $\eta$ increases, the quantum fluctuations become divergent
around the critical point. The finite-frequency scaling function for $\Delta
P$ is calculated dependent on $\eta$ in a log-log relation, as illustrated in
Fig. 4 (b). From the slope of the lines, the critical exponent for the
tricritical point is equal to $\beta_{Q}/\nu=1/2$, but it equals to $1/3$ for
the $2$nd-order critical point. Thus, various finite-frequency scaling laws
are obtained as $(\Delta P)^{2}(\eta,\lambda_{c})\propto\eta^{-1/3}$ for the
$2$nd-order critical point and $(\Delta
P)^{2}(\eta,\lambda_{\texttt{TCP}})\propto\eta^{-1/2}$ for the QTCP,
respectively.

The scaling function in Eq.(8) should be universal for large $\eta$ at the
critical regime, which is independent of $\eta$. According to the critical
exponent $\beta_{Q}/\nu=1/3$ for the second-order phase transition, Fig. 3 (c)
shows the universal scalings of $(\Delta P)^{2}\eta^{1/3}$ as a function of
$(\lambda-\lambda_{c})\eta^{1/\nu}$ for different $\eta$. Remarkably, an
excellent collapse in the critical regime is observed according to the scaling
function in the curve for $\eta=1000,2000,5000$. It demonstrates that the
universal critical exponent is $\nu=3/2$ for the second-order phase
transition, which is the same as that in the quantum Rabi model and Dicke
model [27, 44]. Meanwhile for the QTCP, the universal scaling function
$(\Delta P)^{2}\eta^{1/2}$ as a function of
$(\lambda-\lambda_{\texttt{TCP}})\eta^{1/\nu}$ is shown in Fig. 3 (d). One
observs that curves with the universal critical exponent $\nu=1$ collapse
together. Thus the universal scaling function of $\Delta P$ at the QTCP and
the $2$nd-order critical point are obtained explicitly as

$\displaystyle(\Delta
P)^{2}(\eta,\lambda\to\lambda_{c})\propto\eta^{-1/3}F_{\Delta
P}(|\lambda-\lambda_{c}|\eta^{2/3}),$ (10) $\displaystyle(\Delta
P)^{2}(\eta,\lambda\to\lambda_{\texttt{TCP}})\propto\eta^{-1/2}F_{\Delta
P}(|\lambda-\lambda_{\texttt{TCP}}|\eta).$ (11)

It demontrates that the universal exponent at the QTCP $\nu=1$ is different
from $\nu=3/2$ at the second-order critical point.

To show the universal critical exponent $\nu$ independent of observables, we
investigate the universal scaling of the average photon number $N_{ph}$.
Figs.4 (a) shows $N_{ph}$ as a function of $\eta$ in a log-log scale. The
slope of the line at the QTCP gives the critical exponent $\beta_{Q}/\nu=1/2$,
which is different from $\beta_{Q}/\nu=2/3$ at the $2$nd-order critical point.
Around $2$nd-order critical point, curves of the scaling function for
different scales of $\eta$ collapse into a single curve in Fig. 4(c), which
gives the universal critical exponent $\nu=3/2$. Around the QTCP, we calculate
the universal scaling function $N_{ph}\eta^{1/2}$ dependent on
$(\lambda-\lambda_{\texttt{TCP}})\eta^{1/\nu}$ in Fig. 4(d). An collapse with
$\nu=1$ is achieved for different $\eta$. Thus, for the average photon number
$N_{ph}$, the scaling functions around the $2$nd-order critical point and the
QTCP are obtained, respectively

$\displaystyle
N_{ph}(\eta,\lambda\to\lambda_{c})\propto\eta^{-2/3}F_{N_{ph}}(|\lambda-\lambda_{c}|\eta^{2/3}),$
(12) $\displaystyle
N_{ph}(\eta,\lambda\to\lambda_{\texttt{TCP}})\propto\eta^{-1/2}F_{N_{ph}}(|\lambda-\lambda_{\texttt{TCP}}|\eta).$
(13)

Based on the universal scaling analysis, we have successfully captured various
critical exponents that govern two types of phase transitions. Tab. 1 presents
the critical exponents obtained using the finite-frequency scaling function.
The critical exponent $\beta_{Q}$ varies for different observables $N_{ph}$
and $\Delta P$. In contrast, the critical exponent $\nu$ is a universal
constant that is independent of the physical quantity. Both $\Delta P$ and
$N_{ph}$ predicts the same value of $\nu=3/2$ for the $2$nd-order critical
point. In comparison, the universal critical exponent at the QTCP is equal to
$\nu=1$. It indicates that the QTCP belongs to a nontrivial universality class
with different critical exponents, which goes beyond the second-order
superradiant phase transition in the conventional quantum Rabi and Dicke
models. Recently, a chiral tricritical point has also exhibited a distinct
universality class of phase transitions [11, 37]. It demonstrate that it is
nontrivial to explore quantum critical phenomenons at the tricritical points.

Conclusions –In summary, we have investigated the first and second-order
superradiant phase transitions in the tricritical quantum Rabi model.
According to Landau theory, the ground-state energy is obtained up to sixth-
order perturbation, which displays local minima for the first and second-order
phase transitions. The tricritical point arises at the intersection of the
boundaries for the first and second-order phase transitions. We perform
finite-frequency scaling analysis to calculate the universal scaling of
observables. We find the superradiant phase transition at the tricritical
point belongs to a different universality class with a different universal
critical exponent. The generalization of the well-known quantum Rabi model can
serve as an valuable platform for exploring critical phenomena and more
intricate critical behaviors in few-body systems.

Acknowledgments –This work was supported by NSFC under Grant No.12075040 and
No. 12347101, and Chongqing NSF under Grants No.cstc2020jcyj-msxmX0890.

##  Appendix A Ground-state energy derived by perturbation theory

We employ perturbation theory to derive the ground-state energy in Eq. (4) and
the corresponding coefficients $c_{k}$ in the superradiant phase. The ground-
state energy term of Hamiltonian $H_{0}$ (3) is rewritten as

$\frac{H_{0}}{\Omega}=\frac{1}{4\lambda^{2}}\alpha^{2}+H_{a},$ (14)

where $H_{a}=D+h$ with $D={\alpha}d$ and $\alpha=2\lambda\beta/\sqrt{\eta}$.
The eigenstates of the unperturbed Hamiltonian $h$ are given by
$\left|\varepsilon_{i}\right\rangle$ with corresponding eigenvalues
$\varepsilon_{1}=-1$, $\varepsilon_{2}=0$, and $\varepsilon_{3}=1$. The term
$D$ serves as the perturbation. In the limit $\eta\rightarrow\infty$, $\alpha$
can be treated as a perturbation parameter. So we perform perturbation
expansion up to the order of $\alpha^{6}$. The ground-state wave function can
be expanded as follows:

$\displaystyle|\psi\rangle$ $\displaystyle=|\varepsilon_{1}\rangle+\sum_{m\neq
1}\frac{|\varepsilon_{m}\rangle\langle\varepsilon_{m}|}{E-\varepsilon_{m}}D|\psi\rangle$
(15) $\displaystyle=|\varepsilon_{1}\rangle+G(E)D|\psi\rangle,$

where $G(E)={\textstyle\sum_{m\neq
1}}|\varepsilon_{m}\rangle\langle\varepsilon_{m}|/(E-\varepsilon_{m})$ and
$H_{a}|\psi\rangle=E|\psi\rangle$. This means that the wave function can be
determined through iteration as:

$\displaystyle|\psi\rangle$
$\displaystyle=|\varepsilon_{1}\rangle+G(E)D|\varepsilon_{1}\rangle+G(E)DG(E)D|\varepsilon_{1}\rangle$
(16) $\displaystyle+G(E)DG(E)DG(E)D|\varepsilon_{1}\rangle+\cdots,$

Using the relation $H_{a}|\psi\rangle=(D+h)|\psi\rangle=E|\psi\rangle$, we can
obtain $D|\psi\rangle=(E-\varepsilon_{1})|\psi\rangle$. It yields the ground-
state energy

$E-\varepsilon_{1}=\langle\psi|D|\psi\rangle.$ (17)

By substituting the wave function into the above equation, we obtain the
ground-state energy

$\displaystyle E$
$\displaystyle=\varepsilon_{1}+\langle\varepsilon_{1}|D|\varepsilon_{1}\rangle+\langle\varepsilon_{1}|DG(E)D|\varepsilon_{1}\rangle$
(18)
$\displaystyle+\langle\varepsilon_{1}|DG(E)DG(E)D|\varepsilon_{1}\rangle+\cdots.$

Clearly, the zero-th energy correction is $E^{(0)}=\varepsilon_{1}$. Moreover,
due to the symmetry of the Hamiltonian, the first-order correction is
$\langle\varepsilon_{1}|D|\varepsilon_{1}\rangle=0$. The second-order
correction can be calculated as follows:

$\displaystyle E^{(2)}$
$\displaystyle=\varepsilon_{1}+\left\langle\varepsilon_{1}|DG(E)D|\varepsilon_{1}\right\rangle$
(19)
$\displaystyle=\varepsilon_{1}+\frac{\left|D_{12}\right|^{2}}{E^{(0)}-\varepsilon_{2}}=-1-\alpha^{2}\gamma^{2}.$

Similarly, the fourth-order correction of the ground-state energy is obtained
as

$\displaystyle E^{(4)}$
$\displaystyle=\varepsilon_{1}+\left\langle\varepsilon_{1}|DG(E)D|\varepsilon_{1}\right\rangle$
(20)
$\displaystyle+\left\langle\varepsilon_{1}|DG(E)DG(E)DG(E)D|\varepsilon_{1}\right\rangle$
$\displaystyle=\varepsilon_{1}+\alpha^{2}\frac{\left|D_{12}\right|^{2}}{E^{(2)}-\varepsilon_{2}}$
$\displaystyle+\alpha^{4}\sum_{m\neq 1}\sum_{n\neq 1}\sum_{k\neq
1}\langle\varepsilon_{1}|d\frac{|\varepsilon_{m}\rangle\langle\varepsilon_{m}|}{E_{0}-\varepsilon_{m}}d\frac{|\varepsilon_{n}\rangle\langle\varepsilon_{n}|}{E_{0}-\varepsilon_{n}}d\frac{|\varepsilon_{k}\rangle\langle\varepsilon_{k}|}{E_{0}-\varepsilon_{k}}d|\varepsilon_{1}\rangle$
$\displaystyle=-1+\frac{\alpha^{2}\gamma^{2}}{-1-\alpha^{2}\gamma^{2}}-\frac{1}{2}\alpha^{4}\gamma^{2}.$

Since $\alpha$ is a small value, the second term of the above energy is
approximated as $\alpha^{2}\gamma^{2}(-1+\alpha^{2}\gamma^{2})$. It leads to
the approximated ground-state energy

$E^{(4)}=-1-\alpha^{2}\gamma^{2}+\gamma^{2}(\gamma^{2}-\frac{1}{2})\alpha^{4}.$
(21)

Furthermore, the ground-state energy can be given up to the sixth-order
correction

$\displaystyle E^{(6)}=$
$\displaystyle\varepsilon_{1}+\langle\varepsilon_{1}|DG(E)D|\varepsilon_{1}\rangle$
(22)
$\displaystyle+\langle\varepsilon_{1}|DG(E)DG(E)DG(E)D|\varepsilon_{1}\rangle$
$\displaystyle+\langle\varepsilon_{1}|DG(E)DG(E)DG(E)DG(E)DG(E)D|\varepsilon_{1}\rangle$
$\displaystyle=$
$\displaystyle\varepsilon_{1}+\alpha^{2}\frac{|d_{12}|^{2}}{E^{(4)}-\varepsilon_{2}}$
$\displaystyle+\alpha^{4}\sum_{m\neq 1}\sum_{n\neq 1}\sum_{k\neq
1}\langle\varepsilon_{1}|d\frac{|\varepsilon_{m}\rangle\langle\varepsilon_{m}|}{E^{(2)}-\varepsilon_{m}}d\frac{|\varepsilon_{n}\rangle\langle\varepsilon_{n}|}{E^{(2)}-\varepsilon_{n}}$
$\displaystyle\times
d\frac{|\varepsilon_{k}\rangle\langle\varepsilon_{k}|}{E^{(2)}-\varepsilon_{k}}d|\varepsilon_{1}\rangle$
$\displaystyle+\alpha^{6}\sum_{m\neq 1}\sum_{n\neq 1}\sum_{k\neq 1}\sum_{i\neq
1}\sum_{j\neq
1}\langle\varepsilon_{1}|d\frac{|\varepsilon_{m}\rangle\langle\varepsilon_{m}|}{E^{(0)}-\varepsilon_{m}}d\frac{|\varepsilon_{n}\rangle\langle\varepsilon_{n}|}{E^{(0)}-\varepsilon_{n}}$
$\displaystyle\times
d\frac{|\varepsilon_{k}\rangle\langle\varepsilon_{k}|}{E^{(0)}-\varepsilon_{k}}d\frac{|\varepsilon_{i}\rangle\langle\varepsilon_{i}|}{E^{(0)}-\varepsilon_{i}}d\frac{|\varepsilon_{j}\rangle\langle\varepsilon_{j}|}{E^{(0)}-\varepsilon_{j}}d|\varepsilon_{1}\rangle$
$\displaystyle=$
$\displaystyle\varepsilon_{1}+\frac{\alpha^{2}\gamma^{2}}{E^{(4)}-\varepsilon_{2}}+\alpha^{4}\frac{|d_{12}|^{2}}{(E^{(2)}-\varepsilon_{2})^{2}}\frac{|d_{23}|^{2}}{(E^{(2)}-\varepsilon_{3})^{2}}$
$\displaystyle+\alpha^{6}\frac{|d_{12}|^{3}}{(E^{(0)}-\varepsilon_{2})^{3}}\frac{|d_{23}|^{3}}{(E^{(0)}-\varepsilon_{2})^{3}}.$

With the sixth-order correction, the ground-state energy of the effective
Hamiltonian (14) can be approximately given up to the order of $\alpha^{6}$

$\displaystyle E_{SR}/\Omega=$
$\displaystyle\frac{1}{4\lambda^{2}}\alpha^{2}-1-\alpha^{2}\gamma^{2}+\gamma^{2}(\gamma^{2}-\frac{1}{2})\alpha^{4}$
(23)
$\displaystyle-\frac{1}{4}\gamma^{2}(8\gamma^{4}-7\gamma^{2}+1)\alpha^{6}$
$\displaystyle=c_{1}\alpha^{2}+c_{2}\alpha^{4}+c_{3}\alpha^{6}-1,$

where the coefficients are $c_{1}=1/(4\lambda^{2})-\gamma^{2}$,
$c_{2}=\gamma^{2}(\gamma^{2}-\frac{1}{2})$, and
$c_{3}=-\gamma^{2}(1-7\gamma^{2}+8\gamma^{4})/4$ . The energy is given in
Eq.(4).

Figure 5: Ground-state energy $E_{SR}/\eta$ as a function of the order
parameter $\alpha$ in the superradiant phases of the first-order QPT
($\gamma=0.6<\gamma_{\texttt{TCP}},\lambda=0.82>\lambda_{c}$) in black solid
line, and the second-order QCP
($\gamma=0.8>\gamma_{\texttt{TCP}},\lambda=0.65>\lambda_{c}$) in red dashed
line with $\omega=1$. The energy in the normal phase is listed for
$\gamma=0.1<\gamma_{\texttt{TCP}},\lambda=0.5<\lambda_{c}$ (blue dotted line).

Fig.5 depicts the ground-state energy $E_{SR}/\eta$ as a function of $\alpha$
for various values of $\lambda$ and $\gamma$. By adjusting the transition
ratio $\gamma<\gamma_{\texttt{TCP}}$, $E_{SR}$ locates at three minimal value
of $\alpha$, predicting a first-order phase transition.

## References

  * Sachdev [2011] S. Sachdev,  _Quantum Phase Transitions_, 2nd ed. (Cambridge University Press, Cambridge, 2011).
  * Löhneysen  _et al._ [2007] H. v. Löhneysen, A. Rosch, M. Vojta, and P. Wölfle, Fermi-liquid instabilities at magnetic quantum phase transitions, Rev. Mod. Phys. 79, 1015 (2007).
  * Sondhi  _et al._ [1997] S. L. Sondhi, S. M. Girvin, J. P. Carini, and D. Shahar, Continuous quantum phase transitions, Rev. Mod. Phys. 69, 315 (1997).
  * Griffiths [1975] R. B. Griffiths, Phase diagrams and higher-order critical points, Phys. Rev. B 12, 345 (1975).
  * Belitz  _et al._ [1999] D. Belitz, T. R. Kirkpatrick, and T. Vojta, First order transitions and multicritical points in weak itinerant ferromagnets, Phys. Rev. Lett. 82, 4707 (1999).
  * Belitz and Kirkpatrick [2017] D. Belitz and T. R. Kirkpatrick, Quantum triple point and quantum critical end points in metallic magnets, Phys. Rev. Lett. 119, 267202 (2017).
  * Kaluarachchi  _et al._ [2018] U. S. Kaluarachchi, V. Taufour, S. L. Bud’ko, and P. C. Canfield, Quantum tricritical point in the temperature-pressure-magnetic field phase diagram of ${\mathrm{cetige}}_{3}$, Phys. Rev. B 97, 045139 (2018).
  * Wu  _et al._ [2019] F. Wu, C. Y. Guo, Y. Chen, H. Su, A. Wang, M. Smidman, and H. Q. Yuan, Magnetic field induced antiferromagnetic tricritical points in $\mathrm{Ce}{}_{2}\mathrm{Sb}$ and $\mathrm{Ce}{}_{2}\mathrm{Bi}$, Phys. Rev. B 99, 064419 (2019).
  * Friedemann  _et al._ [2017] S. Friedemann, W. J. Duncan, M. Hirschberger, T. Bauer, R. Kuchler, A. Neubauer, M. Brando, C. Pfleiderer, and F. M. Grosche, Quantum tricritical points in nbfe2, Nature Physics 14, 62 (2017).
  * Xu and Pu [2019] Y. Xu and H. Pu, Emergent universality in a quantum tricritical dicke model, Phys. Rev. Lett. 122, 193201 (2019).
  * Yin  _et al._ [2018] S. Yin, S.-K. Jian, and H. Yao, Chiral tricritical point: A new universality class in dirac systems, Phys. Rev. Lett. 120, 215702 (2018).
  * Kato and Misawa [2015] Y. Kato and T. Misawa, Quantum tricriticality in antiferromagnetic ising model with transverse field: A quantum monte carlo study, Phys. Rev. B 92, 174419 (2015).
  * Cheng  _et al._ [2022] G.-J. Cheng, D. Fallas Padilla, T. Deng, Y.-Y. Zhang, and H. Pu, Chiral quantum phases and tricriticality in a dicke triangle, Quantum Frontiers 1, 2731 (2022).
  * Fallas Padilla and Pu [2023] D. Fallas Padilla and H. Pu, Tricritical dicke model with and without dissipation, Phys. Rev. A 108, 033706 (2023).
  * Greentree  _et al._ [2006] A. D. Greentree, C. Tahan, J. H. Cole, and L. C. Hollenberg, Quantum phase transitions of light, Nature Physics 2, 856 (2006).
  * Hartmann  _et al._ [2006] M. J. Hartmann, F. G. Brandao, and M. B. Plenio, Strongly interacting polaritons in coupled arrays of cavities, Nature Physics 2, 849 (2006).
  * Zhu  _et al._ [2020] C. Zhu, L. Ping, Y. Yang, and G. S. Agarwal, Squeezed light induced symmetry breaking superradiant phase transition, Physical Review Letters 124, 073602 (2020).
  * Dicke [1954] R. H. Dicke, Coherence in spontaneous radiation processes, Phys. Rev. 93, 99 (1954).
  * Lambert  _et al._ [2004] N. Lambert, C. Emary, and T. Brandes, Entanglement and the phase transition in single-mode superradiance, Phys. Rev. Lett. 92, 073602 (2004).
  * Chen  _et al._ [2008] Q.-H. Chen, Y.-Y. Zhang, T. Liu, and K.-L. Wang, Numerically exact solution to the finite-size dicke model, Phys. Rev. A 78, 051801 (2008).
  * Chen and Zhang [2018] X.-Y. Chen and Y.-Y. Zhang, Finite-size scaling analysis in the two-photon dicke model, Phys. Rev. A 97, 053821 (2018).
  * Emary and Brandes [2003] C. Emary and T. Brandes, Chaos and the quantum phase transition in the dicke model, Phys. Rev. E 67, 066203 (2003).
  * Nagy  _et al._ [2010] D. Nagy, G. Kónya, G. Szirmai, and P. Domokos, Dicke-model phase transition in the quantum motion of a bose-einstein condensate in an optical cavity, Phys. Rev. Lett. 104, 130401 (2010).
  * zhang  _et al._ [2021] X. zhang, C. Yu, W. Zemao, W. Juan, F. Jijie, D. Shujin, and W. Haibin, Observation of a superradiant quantum phase transition in an intracavity degenerate fermi gas, Science 373, 1359 (2021).
  * Ashhab [2013] S. Ashhab, Superradiance transition in a system with a single qubit and a single oscillator, Phys. Rev. A 87, 013826 (2013).
  * Hwang  _et al._ [2015] M.-J. Hwang, R. Puebla, and M. B. Plenio, Quantum phase transition and universal dynamics in the rabi model, Phys. Rev. Lett. 115, 180404 (2015).
  * Liu  _et al._ [2017] M. Liu, S. Chesi, Z.-J. Ying, X. Chen, H.-G. Luo, and H.-Q. Lin, Universal scaling and critical exponents of the anisotropic quantum rabi model, Phys. Rev. Lett. 119, 220601 (2017).
  * Chen  _et al._ [2020a] X.-Y. Chen, Y.-Y. Zhang, L. Fu, and H. Zheng, Generalized coherent-squeezed-state expansion for the super-radiant phase transition, Phys. Rev. A 101, 033827 (2020a).
  * Lü  _et al._ [2018] X.-Y. Lü, L.-L. Zheng, G.-L. Zhu, and Y. Wu, Single-photon-triggered quantum phase transition, Phys. Rev. Appl. 9, 064006 (2018).
  * Chen  _et al._ [2021] X. Chen, Z. Wu, M. Jiang, X.-Y. Lü, X. Peng, and J. Du, Experimental quantum simulation of superradiant phase transition beyond no-go theorem via antisqueezing, Nat. Commun. 12, 1 (2021).
  * Cai and et al. [2021] M. L. Cai and et al., Observation of a quantum phase transition in the quantum rabi model with a single trapped ion, Nat. Commun. 12, 1126 (2021).
  * Hwang and Plenio [2016] M.-J. Hwang and M. B. Plenio, Quantum phase transition in the finite jaynes-cummings lattice systems, Phys. Rev. Lett. 117, 123602 (2016).
  * Jiang  _et al._ [2021] X. Jiang, B. Lu, C. Han, R. Fang, M. Zhao, Z. Ma, T. Guo, and C. Lee, Universal dynamics of the superradiant phase transition in the anisotropic quantum rabi model, Phys. Rev. A 104, 043307 (2021).
  * Chen  _et al._ [2020b] X.-Y. Chen, Y.-F. Xie, and Q.-H. Chen, Quantum criticality of the rabi-stark model at finite frequency ratios, Phys. Rev. A 102, 063721 (2020b).
  * Shen  _et al._ [2017] L.-T. Shen, Z.-B. Yang, H.-Z. Wu, and S.-B. Zheng, Quantum phase transition and quench dynamics in the anisotropic rabi model, Phys. Rev. A 95, 013819 (2017).
  * Zhang  _et al._ [2021] Y.-Y. Zhang, Z.-X. Hu, L. Fu, H.-G. Luo, H. Pu, and X.-F. Zhang, Quantum phases in a quantum rabi triangle, Phys. Rev. Lett. 127, 063602 (2021).
  * Fallas Padilla  _et al._ [2022] D. Fallas Padilla, H. Pu, G.-J. Cheng, and Y.-Y. Zhang, Understanding the quantum rabi ring using analogies to quantum magnetism, Phys. Rev. Lett. 129, 183602 (2022).
  * [38] L.-J. Li, L.-L. Feng, J.-H. Dai, and Y.-Y. Zhang, Quantum rabi hexagonal ring in an artificial magnetic field, arXiv:2304.01535 .
  * Masson  _et al._ [2017] S. J. Masson, M. D. Barrett, and S. Parkins, Cavity qed engineering of spin dynamics and squeezing in a spinor gas, Phys. Rev. Lett. 119, 213601 (2017).
  * Krauth  _et al._ [1992] W. Krauth, M. Caffarel, and J.-P. Bouchaud, Gutzwiller wave function for a model of strongly interacting bosons, Phys. Rev. B 45, 3137 (1992).
  * Buonsante  _et al._ [2009] P. Buonsante, F. Massel, V. Penna, and A. Vezzani, Gutzwiller approach to the bose-hubbard model with random local impurities, Phys. Rev. A 79, 013623 (2009).
  * Fisher and Barber [1972] M. E. Fisher and M. N. Barber, Scaling theory for finite-size effects in the critical region, Phys. Rev. Lett. 28, 1516 (1972).
  * Botet  _et al._ [1982] R. Botet, R. Jullien, and P. Pfeuty, Size scaling for infinitely coordinated systems, Phys. Rev. Lett. 49, 478 (1982).
  * Liu  _et al._ [2009] T. Liu, Y.-Y. Zhang, Q.-H. Chen, and K.-L. Wang, Large-$n$ scaling behavior of the ground-state energy, fidelity, and the order parameter in the dicke model, Phys. Rev. A 80, 023810 (2009).