# PFL-MoE: Personalized Federated Learning Based on Mixture of Experts

1st Binbin Guo
4th Weigang Wu  School of Comp. Sci. & Eng.
Sun Yat-sen University
Guang Zhou, China
<EMAIL_ADDRESS>School of Comp. Sci. & Eng.
Sun Yat-sen University
Guang Zhou, China
<EMAIL_ADDRESS>2nd Yuan Mei
5thYe Yin  School of Comp. Sci. & Eng.
Sun Yat-sen University
Guang Zhou, China
<EMAIL_ADDRESS>Interactive Entertainment Group
Tencent Inc.
Shenzhen, China
<EMAIL_ADDRESS>3rdDanyang Xiao
6thHongli Chang  School of Comp. Sci. & Eng.
Sun Yat-sen University
Guang Zhou, China
<EMAIL_ADDRESS>Interactive Entertainment Group
Tencent Inc.
Shenzhen, China
<EMAIL_ADDRESS>
###### Abstract

Federated learning (FL) is an emerging distributed machine learning paradigm
that avoids data sharing among training nodes so as to protect data privacy.
Under coordination of the FL server, each client conducts model training using
its own computing resource and private data set. The global model can be
created by aggregating the training results of clients. To cope with highly
non-IID data distributions, personalized federated learning (PFL) has been
proposed to improve overall performance by allowing each client to learn a
personalized model. However, one major drawback of a personalized model is the
loss of generalization. To achieve model personalization while maintaining
generalization, in this paper, we propose a new approach, named PFL-MoE, which
mixes outputs of the personalized model and global model via the MoE
architecture. PFL-MoE is a generic approach and can be instantiated by
integrating existing PFL algorithms. Particularly, we propose the PFL-MF
algorithm which is an instance of PFL-MoE based on the freeze-base PFL
algorithm. We further improve PFL-MF by enhancing the decision-making ability
of MoE gating network and propose a variant algorithm PFL-MFE. We demonstrate
the effectiveness of PFL-MoE by training the LeNet-5 and VGG-16 models on the
Fashion-MNIST and CIFAR-10 datasets with non-IID partitions.

###### Index Terms:

Federated Learning, Personalization, Mixture of Experts, Fine-tuning,
Distributed Machine Learning

##  I Introduction

In the past few years, we have witnessed the rapid development of deep
learning in artificial intelligence (AI) applications, such as computer
vision, natural language processing, and recommendation systems. As is known
to all, the success of deep learning technologies relies on massive data set
which are usually the collections of data from different organizations,
devices, or users. And distributed machine learning (DML) [1], which may
employ a large number of worker nodes to collaboratively conduct training, has
been widely used nowadays to train large scale neural network models on large
volume data set. However, in many scenarios, data is in fact distributed among
different clients (owned by different users) and is sensitive to privacy. Such
data cannot be collected and shared with any other nodes. On the other hand,
as the storage and computational capabilities of edge devices grow, it is
becoming increasingly attractive to perform learning directly on the edge [2].
Unfortunately, it is difficult for a client to learn an effective model using
only its own data, because the data set at a single client is usually not
large enough. Federated Learning (FL) allows a large number of clients (e.g.,
mobile phones and IoT devices) to learn a global model together without data
sharing. The global model is created by repeatedly averaging model updates
from small subsets of clients. More specifically, each round of FL includes
three basic phases: (1) the FL server delivers the global model to each
participating client; (2) the clients conduct training based on its private
data and return the updated models to the server; and (3) the FL server gets
the latest global model by aggregating the uploaded models. As the training
converges, the final global model can achieve performance similar to the model
trained via pooled and shared data.

One of major challenges of FL is the statistical diversity among data owned by
different clients. The distribution of data across clients is natural
inconsistency and non-IID (IID, independent and identically distributed), and
the amount of data is also highly biased. The non-IID characteristic may
largely affect the convergence behaviors of the global model trained, because
non-IID data may cause the model parameters fluctuates largely and delay or
even destroy the convergence. It will be even worse if privacy-preserving
mechanisms, e.g., differential privacy [3], are taken into consideration.
Besides, different clients have great accuracy differences under the same
global model. In some clients, the global model is even worse than the locally
trained model.

Personalization of the global model is necessary to handle the statistic
heterogeneity of the distributed data. In personalized federated learning
(PFL), each client learns a personalized model suitable for its own data
distribution, which is more flexible than a single global model. Yu et al. [4]
has investigated various technologies for adapting the federated global model
to an individual client, including fine-tuning [5], multi-task learning [6],
and knowledge distillation [7]. These PFL algorithms can effectively improve
the model performance for individual participants. Almost all personalized
models outperform the corresponding locally trained models, which greatly
motivates clients to participate in federated learning. However,
personalization is always accompanied by the loss of generalization, which is
manifested in the increases of generalization error of personalized model. For
example, a client has only a small amount of data, and the personalized model
is likely to overfit the local data. How to balance personalization and
generalization is the current research focus of personalized federated
learning.

Most of existing PFL algorithms attempt to find a single model that is both
personalized and generalized. Alternatively, FL+DE [8] combines two models to
improve accuracy based on the mixture of experts (MoE) [9]. MoE is one of the
most popular combining methods. It is established based on the divide-and-
conquer principle in which the problem space is divided between a few neural
network experts, supervised by a gating network [10]. In FL+DE, all clients
collaborate to build a general global model, but maintain private, domain-
adapted expert models. In each client, the global model and the private model
are regarded as global expert and domain expert respectively and their outputs
are mixed by the gating network. The predictions of the two models are much
more accurate than those of the single global model.

Inspired by FL+DE, we propose a personalized federated learning approach PFL-
MoE that considers the personalized model as local expert and combines it with
the federated global model through MoE architecture. PFL-MoE is a generic
approach and can be instantiated by existing PFL algorithms. Particularly, we
propose PFL-MF algorithm, which is an instance of PFL-MoE based on the freeze-
based PFL (PFL-FB) algorithm [4]. PFL-FB is an effective algorithm adapting
the federated global model to an individual client.

Moreover, we find that the MoE architecture cannot work well with complex
input data, such as the color images, because it is difficult for the gating
network to extract useful features from high-dimensional input data for
decision-making. To enhance the decision-making ability of MoE gating network,
we modify the model structure of PFL-MF and propose an extended algorithm PFL-
MFE. More precisely, PFL-MFE uses abstract features extracted by the base
layer as the input of MoE gating network.

We conduct extensive experiments to evaluate the performance of our algorithms
and compare them with existing ones. We use three combinations of dataset and
model: Fashion-MNIST + LeNet-5, CIFAR10 + LeNet-5, and CIFAR-10 + VGG-16. In
various non-IID experimental settings, the proposed algorithms achieve higher
global accuracy while the same or even better local test accuracy than PFL-FB.
The results show our approach combines the advantages of the personalized
model and the global model.

Contributions:

  * •

We propose a novel PFL approach, namely, PFL-MoE, that can achieve both
personalization and generalization. PFL-MoE is generic and can be instantiated
using existing FL algorithms.

  * •

Based on our PFL-MoE approach, we propose the PFL-MF algorithm and its
extension, the PFL-MFE algorithm. The former is an instance of PFL-MoE and the
later improves PFL-MF by modifying the MoE architecture so as to achieve
better decision-making ability.

  * •

We conduct experiments to evaluate our approach. An overall evaluation of PFL-
MF and PFL-MFE algorithms is provided. The experiment results confirm the
advantage of our design.

The rest of the paper is organized as follows. Section II reviews related
works on personalization techniques for federated learning. Section III
describes the detailed design of the PFL-MoE approach, and the instance
algorithms PFL-MF and PFL-MFE are presented in Section IV. Section V reports
the performance evaluation results. Finally, Section VI concludes the paper.

##  II Related Works

###  II-A Classical Federated Learning

The first FL system is developed by Google [11, 12], which selects a sample of
available smartphones to update the language prediction model based on the
edge server architecture, and has got a good result. However, there are still
many challenges in federated learning, including privacy concerns,
communication cost, systems heterogeneity and statistical heterogeneity [13,
14]. Although federated learning only communicates model parameters, studies
have shown that monitoring model updates can reversely derive data, making
privacy protection an important issue [15]. A considerable amount of
literature has been published on privacy leakage risks and protection methods
[16]. The FL server coordinates the participants to iterative training a
global model, where uploading and downloading model parameters require a lot
of communication overhead. Some studies aim to reduce communication costs in
FL [12]. In federated settings, all edge devices can be clients. In most
cases, these devices have fewer network resources and cannot keep online all
the time [12]. Moreover, the software and hardware resources and computing
power of each client vary greatly. These issues requires the federated
learning framework to coordinate and allocate the clients effectively [17].

###  II-B Handling Non-IID and Personalization

Given the distributed nature of the clients, the statistics of the data across
them are likely to differ significantly, which is another big challenge for FL
[14]. A large number of experiments have evaluated the impact of non-IID on FL
[18]. Simultaneously, various non-IID data construction methods [18] and the
non-IID datasets from the real world [19] are emerging. Li et al. [20]
provided the first proof of the convergence of FedAvg[11], widely used in
federated learning, under non-IID data distributions. Zhao et al. [21]
proposed a method of sharing part of the global data to alleviate data
dissimilarity among the clients, so as to improve the convergence rate and
performance of the global model. In FedProx [17], the proximal term was added
to each client’s optimization goal to limit the drift of its update so that
the algorithm has better robustness and convergence stability than FedAvg.

Alternatively, personalized federated learning [22] starts from the incentive
of the clients participating in federated learning, and turns the goal to
improve the accuracy of the personalized models of the clients with non-IID
data distributions, which improves the overall performance effectively. A
large and growing body of literature has investigated to achieve
personalization through various methods:

##### Fine-turning and meta-learning

In Wang et al.’s work [5], on each client, the personalized model was fine-
tuned from the federated global model (PFL-FT), but the used of the model is
depends on whether or not it outperforms the global model. In his image
classification experiment, most clients can get a better personalized model,
and the rest of the clients continue to use the global model. Jiang et al.
[23] observed that the FedAvg algorithm can be interpreted as a meta learning
algorithm (MAML) [24]. This provides a good explanation for fine-tuning can
achieve good personalization that FedAvg can get a good initial model for the
clients which achieve high precision with only a little local training. Also
based on the MAML algorithm, Fallah et al. proposed a variant of FedAvg named
Per-FedAvg [24] which turns the goal into learning a better initial model for
each client and provides theoretical guarantees. Furthermore, pFedMe [25]
updates both the personalized and global models by solving the bi-level
optimization problem instead of using the global model as initialization,
which is more flexible than Per-FedAvg.

##### Multi-task

Smith et al. [6] observed that multi-task learning is naturally suitable for
federated learning. They treated each client as a individual task in multi-
task learning literature, and investigated a system-aware approach called
MOECHA that takes into account communication costs, straggles, and fault
tolerance. Not limited to convex models, Corinzia et al. [26] proposed the
VIRTUAL algorithm to solve generic non-convex models for federated multi-task
learning in the scenarios where the data is highly Non-IID.

##### Knowledge distillation

A new federated learning framework FedMD [7] was proposed, allowing
participants to independently design their own models. FedMD achieves the
knowledge transmission between each client through transfer learning and
knowledge distillation.

##### Mixing models

In Hanzely et al.’s work [27], a new federated learning optimization goal was
defined to seek a mixture of global and local models. The method controls the
mixing degree of the local and global models through $\lambda$. When
$\lambda=0$, it means each client is trained locally, and when
$\lambda=\infty$, it is equivalent to FedAvg algorithm. When taking
$\lambda>0$, it means personalization is encouraged. They further theorize
about the definition and come up with an algorithmic independent theory.
Furthermore, APFL [28] determines each client’s contribution through
optimization, with each client contributing to the global model while training
its own local model. APFL is also a communication-effective algorithm. It is
proved that the mixing of the local model and global model can reduce the
generalization error. Google researchers studied a systematic, individualized
learning theory and analyzes three kinds of individualized methods [29],
including user clustering, data interpolation, and model interpolation. Model
interpolation is obtained from the weighted average of parameter of the local
model and global model, and its performance is proved by theory and
experiment. FL+DE [8] uses privacy-preserving FL to train a generic model and
mixes it with each user’s private domain model using the MoE architecture.

##  III PFL-MoE

PFL-MoE is a generic approach for personalization, which mainly contains the
following three stages:

  * •

Federated Learning Stage: Following the traditional FL framework, each client
participates in FL training. The global model is created by repeatedly
aggregating model updates from small subsets of clients. This stage is
agnostic about the aggregation methods or privacy-preserving techniques of
federated learning. It only needs the final global model for the next
personalization.

  * •

Personalization Stage: Each client downloads the latest global model from the
FL server, and then conducts local adaption to get a personalized model based
on the global model and local data.

  * •

Mixing Stage: Via the previous two stages, the client gets two models, one is
the personalized model, and the other is the global model. Based on the MoE
architecture, the parameters of the gating network are trained to combine the
models and make them work together.

###  III-A Federated Learning

In federated learning, we consider that the number of clients participating in
training is N. We define the model structure of the global model as $M_{G}$,
and define the parameter of the global model as $\theta$ ($d_{1}$-dimensional
vector). At the beginning of training, the global model structure $M_{G}$ has
been determined by the central FL server. The goal of federated learning is to
find a $\theta\in\mathbb{R}^{d_{1}}$ that makes the average expected loss of
all clients minimal. We define $F:\mathbb{R}^{d_{1}}\rightarrow\mathbb{R}$ as
the global loss of FL, so the learning goal of FL is the following formula:

$\min_{\theta\in\mathbb{R}^{d_{1}}}F(\theta)=\frac{1}{N}\sum_{i=1}^{N}f_{i}(\theta)$
(1)

Since the data of each client follow a different distribution, we define the
data distribution of the client $i$ as $\mathcal{D}_{i}$. We define the
expected loss over $\mathcal{D}_{i}$ of the client $i$ as $f_{i}$. In
particular, we assume that the learning is a supervised learning task. At this
time, $f_{i}$ can be written as:

$f_{i}(\theta)=\mathbb{E}_{(x,y)\sim\mathcal{D}_{i}}[L_{i}(M_{G}(\theta;x),y)]$
(2)

where $(x,y)$ is a pair of data composed of the input and the corresponding
label, following the distribution $\mathcal{D}_{i}$. $L_{i}$ is the loss
function of the client $i$, usually, it is uniform in each client. E.g., in a
classification task using a neural network, it is the cross entropy loss
function.

###  III-B Personalization

In PFL, the optimization goal is the same as individual learning. However, in
individual learning, due to the insufficient local data, it is difficult for a
client to estimate its local data distribution, which leads to a large
generalization error. Based on the federated global model, it is easier for
individuals to obtain models with better generalization capabilities than
individual learning. In particular, we chose PFL-FT, a local adaptation PFL
algorithm based on fine-tuning, to formally describe the details of the
personalization stage. The goal is to find a good model parameter for each
client and we define $F:\mathbb{R}^{Nd1}\rightarrow\mathbb{R}$ as the global
loss:

$\min_{\theta_{1},\theta_{2},...,\theta_{N}\in\mathbb{R}^{d_{1}}}F(\theta_{1},\theta_{2},...,\theta_{N})=\frac{1}{N}\sum_{i=1}^{N}f_{i}(\theta_{i})$
(3)

$f_{i}(\theta_{i})=\mathbb{E}_{(x,y)\sim\mathcal{D}_{i}}[L_{i}(\widehat{y},y)]$
(4)

$\widehat{y}=M_{G}(\theta_{i};x)$ (5)

where the $\theta_{i}$ is what we want for the client $i$ through PFL, and the
$\widehat{y}$ is the output pseudo label corresponding to x.

In PFL-FT, the initial value of $\theta_{i}$ is the global model parameter
$\theta$, and the $\theta_{i}$ is updated by stochastic gradient descent (SGD)
with a relatively small learning rate:

$\theta_{i}=\theta_{i}-\alpha\cdot\triangledown{f_{i}(\theta_{i})}$ (6)

where $\alpha$ is the learning rate of the $\theta_{i}$.

###  III-C Mixing

The proposed PFL-MoE mixes the output of the personalized model and the global
model using MoE architecture. MoE architecture is a method of ensemble
learning. Different from the typical ensemble learning methods, the individual
learners of MoE are trained for different subtasks, so its diversity can be
guaranteed. In the MoE architecture, experts are independent on each other,
and there is no limit that each expert must have the same mode structure [8].

Algorithm 1 PFL-MoE

0: global model structure $M_{G}$, and latest global model parameter $\theta$,
client’s data $D_{i}$, $\theta_{i}$’s learning rate $\alpha$, $w_{i}$’s
learning rate $\beta$, local adaptation epochs $E$, loss function $L$

0: personalized model parameter $\theta_{i}$, gating network parameter $w_{i}$

1: # Initialize client’s personalized model parameter $\theta_{i}$ with global
$\theta$

2: $\theta_{i}\leftarrow\theta$

3: divide $D_{i}$ to $D_{i}^{per}$, $D_{i}^{gate}$

4: for epoch $e=1$ to $E$ do

5: # Local adaptation

6: for batch $(x,y)\subset D_{i}^{per}$ do

7: $\widehat{y}\leftarrow M_{C}(\theta_{i};x)$

8:
$\theta_{i}\leftarrow\theta_{i}-\alpha\cdot\triangledown_{\theta_{i}}{L(\widehat{y},y)}$
# if using fine-tuning

9: end for

10: # Mixing two experts

11: for batch $(x,y)\subset D_{i}^{gate}$ do

12: $g\leftarrow sigmoid(G(w_{i};x))$

13: $\tilde{y}\leftarrow g\cdot M_{G}(\theta;x)+(1-g)\cdot
M_{G}(\theta_{i};x)$

14: $w_{i}\leftarrow w_{i}-\beta\cdot\triangledown_{w_{i}}{L(\tilde{y},y)}$

15: end for

16: end for

More formally, in MoE architecture, the mxied output
$\tilde{y}=\sum_{i=1}^{n}G(x)_{i}\cdot M_{i}(x)$, where
$\sum_{i=1}^{n}G(x)_{i}=1$ and $M_{i},i=1,...,n$ are $n$ experts (neural
networks). Specifically, the gating network $G$ produces a probability
distribution on $n$ experts, and the final output is the weighted sum of all
experts. In our method, there are $n=2$ experts: the personalized model is
viewed as the local expert, while the global model is viewed as the global
expert. Their outputs are mixed by the gating network. The gating is a linear
neural network, and can also be trained with SGD.

We define the gating network as $G$. For client $i$, we define its parameter
of $G$ as $w_{i}$ ($d_{2}$-dimensional vector), and the loss function of
gating $p_{i}$ is given by Eq. 7:

$p_{i}(w_{i})=\mathbb{E}_{(x,y)\sim\mathcal{D}_{i}}[L_{i}(\tilde{y},y)]$ (7)

where $p_{i}:\mathbb{R}^{d_{2}}\rightarrow\mathbb{R}$, and the mixed pseudo
label $\tilde{y}$ is given by Eq. 8 and Eq. 9 when the input is $x$:

$g=sigmoid(G(w_{i};x))$ (8)

$\tilde{y}=g\cdot M_{G}(\theta;x)+(1-g)\cdot M_{G}(\theta_{i};x)$ (9)

where the $G$ is a linear neural network. $sigmoid$ is the activation function
of the model output, and the term $g$ represents the mixing ratio of the
expert models. For details, the number of the input neurons of $G$ is equal to
the dimension of $x$, while the number of the output neurons of $G$ is equal
to the number of experts. Particularly, similar to Peterson et al. [8], we set
the number of output neurons of $G$ to 1, so that $g$ is a 1-dimensional
vector. At this setting, $g$ is the weight of the global expert, while $1-g$
is that of local expert. For the client $i$, to minimize $p_{i}(w_{i})$, the
gating parameter $w_{i}$ can be updated using SGD:

$w_{i}=w_{i}-\beta\cdot\triangledown{p_{i}(w_{i})}$ (10)

where the term $\beta$ refers to the learning rate of $w_{i}$. Alg.1 shows the
details of the PFL-MoE.

##  IV PFL-MF and PFL-MFE

PFL-FB [4] is a local adaption PFL algorithm based on freeze-base that freezes
the base layer of the federated model and fine-tunes only the top layers. PFL-
FB has the most significant personalization effect on the comparison of
several methods for adapting federated global model to individual client.

Following the three steps of PFL-MoE, we propose the PFL-MF algorithm by using
the local expert generated by the PFL-FB algorithm. PFL-FB is a variant of
PFL-FT [5], where the global model structure $M_{G}$ is divided into two
parts: feature extractor $M_{E}$ and classifier $M_{C}$. We define
$\theta_{E}$ and $\theta_{C}$ as the parameter of $M_{E}$ and $M_{C}$
respectively. In PFL-MF, assume that the $\theta$ is the parameter of global
model, for client $i$, $(\theta_{E},\theta_{C_{i}})$ is initialized with
$\theta$, where the term $\theta_{C_{i}}$ is the personalized classifier
parameter of the client and the $\theta_{E}$ is shared to all clients. In
personalization stage, the $\theta_{C_{i}}$ is updated by:

$\theta_{C_{i}}=\theta_{C_{i}}-\alpha\cdot\triangledown{f_{i}(\theta_{E},\theta_{C_{i}})}$
(11)

$f_{i}(\theta_{E},\theta_{C_{i}})=\mathbb{E}_{(x,y)\sim\mathcal{D}_{i}}[L_{i}(\widehat{y},y)]$
(12)

where the pseudo label $\widehat{y}$ corresponding to x is given by:

$a=M_{E}(\theta_{E},x)$ (13)

$\widehat{y}=M_{C}(\theta_{C};a)$ (14)

where $a$ is the activations between $M_{E}$ and $M_{C}$.

In the mixing stage, $w_{i}$ is updated by Eq. 10, and the gating loss
$p_{i}(w_{i})$ is defined in Eq. 7. In the PFL-MF algorithm, different from
Eq. 9, the mixed pseudo label $\tilde{y}$ corresponding to $x$ is written as:

$\tilde{y}=g\cdot M_{C}(\theta_{C};a)+(1-g)\cdot M_{C}(\theta_{C_{i}};a)$ (15)

Through experiments on the Fashion-MNIST dataset, we observe that PFL-MF can
achieve good personalization and maintain good generalization capability.
However, due to the limitation of the simple linear network, the gating works
not well when the input is more complex high-dimensional data.

To address this issue, we propose PFL-MFE algorithm that using features as the
input of gating. In convolutional neural networks, it is recognized that the
base layer is responsible for capturing general features, while the deeper
part is responsible for classification. Intuitively, the features are a better
choice for the gating input compared to raw data. Based on this idea, in the
PFL-MFE algorithm, we modify the model structure of the PFL-MF algorithm, and
use the activations $a$ as the gating input:

$g=sigmoid(G(w_{i};a))$ (16)

at this time, the number of the input neurons of $G$ is equal to the dimension
of $a$ instead of $x$. Alg. 2 describes the implementation details of the PFL-
MF and PFL-MFE algorithms for client $i$.

Algorithm 2 PFL-MF(E) Algorithm

0: global extractor model structure $M_{E}$ and its parameter $\theta_{E}$,
global classifier model structure $M_{C}$ and its parameter $\theta_{C}$,
client’s data $D_{i}$, $\theta_{C_{i}}$’s learning rate $\alpha$, $w_{i}$’s
learning rate $\beta$, local adaptation epochs $E$, loss function $L$, if
variant flag $v$

0: personalized classifier parameter $\theta_{C_{i}}$, gating network
parameter $w_{i}$

1: # Initialize client’s personalized classifier parameter $\theta_{C_{i}}$
with global $\theta_{C}$

2: $\theta_{C_{i}}\leftarrow\theta_{C}$

3: divide $D_{i}$ to $D_{i}^{per}$, $D_{i}^{gate}$

4: for epoch $e=1$ to $E$ do

5: # Local adaptation

6: for batch $(x,y)\subset D_{i}^{per}$ do

7: $\widehat{y}\leftarrow M_{C}(\theta_{C};M_{E}(\theta_{E};x))$

8:
$\theta_{C_{i}}\leftarrow\theta_{C_{i}}-\alpha\cdot\triangledown_{\theta_{C_{i}}}{L(\widehat{y},y)}$

9: end for

10: # Mixing two experts

11: for batch $(x,y)\subset D_{i}^{gate}$ do

12: $a\leftarrow M_{E}(\theta_{E};x)$

13: if $v$ then

14: $g\leftarrow sigmoid(G(w_{i};a))$ # PFL-MFE

15: else

16: $g\leftarrow sigmoid(G(w_{i};x))$ # PFL-MF

17: end if

18: $\tilde{y}\leftarrow g\cdot M_{C}(\theta_{C};a)+(1-g)\cdot
M_{C}(\theta_{C_{i}};a)$

19: $w_{i}\leftarrow w_{i}-\beta\cdot\triangledown_{w_{i}}{L(\tilde{y},y)}$

20: end for

21: end for

Figure 1: The local test accuracy and global test accuracy in Local for 100
clients. Each box contains 100 test accuracy of 100 clients for each specific
$\alpha$.

##  V Experimental Evaluation

###  V-A Experimental Setup

####  V-A1 Datasets and models

In our experiments111All code and results can be found at
https://github.com/guobbin/PFL-MoE., we use two image recognition datasets to
conduct model training: Fashion-MNIST [30] and CIFAR-10 [31]. With two network
models trained, we have three combinations: Fashion-MNIST + LeNet-5 [32],
CIFAR-10 + LeNet-5, and CIFAR-10 + VGG-16 [33]. Fashion-MNIST is an
alternative dataset of MNIST. It contains 28$\times$28 grayscale images of the
front side of 70,000 different products in 10 categories, divided into
60000/10000 training/testing datasets. CIFAR-10 is a dataset of $32\times 32$
color images closer to universal objects, with 3 channels and a
training/testing dataset division of 50000/10000. LeNet-5 contains two
convolutional layers and three fully connected layers. We resize the images of
the datasets to $32\times 32$ uniformly, and due to the difference in the
number of image channels, the corresponding number of the LeNet-5 model
parameters are 61706 and 62006, respectively. VGG-16 is a relatively deep
neural network, including 13 convolutional layers and 3 fully connected
layers, with a total of 15,245,130 parameters.

####  V-A2 Non-IID Data

Similar to Hsu et al. [18], we use Dirichlet distribution with $\alpha$ to
simulate the non-IID data distributions. $\alpha>0$ is a concentration
parameter controlling the identicalness among clients. The larger the
$\alpha$, the higher the similarity between the distributions, and we chose
$\alpha=[0.5,0.9,2]$ to generate three non-IID data distributions for
experiments. For each $\alpha$, we split the training set and allocate images
from each class to 100 clients.

####  V-A3 Testing

Each client has two types of tests, including local test and global test.
Local test uses the test set of the same distribution as the training set of
the same client. In contrast, global test uses the global test set (10000 test
images). Following the local test method of Yu et al. [4], for each client, we
calculate the client’s per-class accuracy on the global test set, multiply it
by the corresponding class’s ratio in the training set of the client, and sum
up the resulting values. The accuracy of the local test and global test is
used as a measure of personalization and generalization.

####  V-A4 Baselines and PFL-MoE

To verify the effectiveness of the proposed personalization algorithms PFL-MF
and PFL-MFE, we compare them with three baselines: Local, FedAvg, and PFL-FB.
The detailed settings of each algorithm are as follows:

TABLE I: The average value of local test accuracy of all clients in three baselines and proposed algorithms. Bold means the best in all methods.  | non-IID $\alpha$ | Local(%) | FedAvg(%) | PFL-FB(%) | PFL-MF(%) | PFL-MFE(%)
---|---|---|---|---|---|---
Fashion-MNIST & LeNet5 | 0.5 | 84.87 | 90 | 92.84 | 92.85 | 92.89
0.9 | 82.23 | 90.31 | 91.84 | 92.02 | 92.01
2 | 78.63 | 90.5 | 90.47 | 90.97 | 90.93
CIFAR-10 & LeNet5 | 0.5 | 65.58 | 68.92 | 77.46 | 75.49 | 77.23
0.9 | 61.49 | 70.7 | 74.7 | 74.1 | 74.74
2 | 55.8 | 72.69 | 72.5 | 73.24 | 73.44
CIFAR-10 & VGG-16 | 0.5 | 52.77 | 88.16 | 91.92 | 90.63 | 91.71
0.9 | 45.24 | 88.45 | 91.34 | 90.63 | 91.18
2 | 34.2 | 89.17 | 90.4 | 90.15 | 90.4
TABLE II: The average value of global test accuracy of all clients. Bold means
the best in all personalization algorithms. For FedAvg, all clients have the
same global model, and they use the same global test set for the global test,
so their global test accuracy is the same. The value is exactly the average
local test accuracy of above local test, due to the sum of the proportions of
each category of all clients is the same as the global test set.
| non-IID $\alpha$ | Local(%) | FedAvg(%) | PFL-FB(%) | PFL-MF(%) | PFL-MFE(%)
---|---|---|---|---|---|---
Fashion-MNIST & LeNet5 | 0.5 | 57.77 | 90 | 83.35 | 85.45 | 85.3
0.9 | 65.28 | 90.31 | 85.91 | 87.69 | 87.67
2 | 71.06 | 90.5 | 87.77 | 89.37 | 89.18
CIFAR-10 & LeNet5 | 0.5 | 28.89 | 68.92 | 54.28 | 62.33 | 58.27
0.9 | 32.1 | 70.7 | 59.93 | 65.78 | 64.13
2 | 35.32 | 72.69 | 66.06 | 69.79 | 69.78
CIFAR-10 & VGG-16 | 0.5 | 21.53 | 88.16 | 82.39 | 85.81 | 84.05
0.9 | 22.45 | 88.45 | 82.62 | 88.15 | 87.9
2 | 21.27 | 89.17 | 88.77 | 89.3 | 89.3

##### Local

Each client’s local model was trained with its local training set for 300
epochs with the regularization factor $\lambda=0.0005$, batch size $B=64$. The
optimizer is SGD with the momentum $m=0.9$, learning rate $\eta=0.1$, and
learning rate decay $\gamma=0.1$ every 100 epochs or Adam. Each Local
experiment chose the better one as the final experimental result. The results
of the Local are shown in Fig. 1.

##### FedAvg

We trained the global model for 1000 rounds, following McMahan et al. [11],
each round randomly selected $c=0.1$ rate of the clients for training with the
learning rate $\eta=0.01$, momentum $m=0.5$, local batch size $b=10$, and
local epoch $e=5$. We saved the model parameters that performed best on the
global test set in these 1,000 rounds for the local adaptation of the
following personalization algorithms. The results of FedAvg algorithm are
shown in Fig.2.

Figure 2: Global test accuracy of FedAvg, 1000 rounds, participate rate 0.1,
100 clients.

##### Personalization algorithms

Each client trained 200 epochs for local adaption using the same SGD optimizer
as the Local baseline. In PFL-FB experiments, the convolutional layers of
LeNet-5 and VGG-16 were fixed, and the top layers were trained with the
learning rate $\alpha=0.001$ and $\alpha=0.005$, respectively. In PFL-MF
experiments, the optimizer of gating network was SGD with the learning rate
$\beta=0.001$. The input layer of gating network has $32\times 32\times
1=1024$ and $32\times 32\times 3=3072$ neurons for Fashion-MNIST and CIFAR-10,
respectively. But in PFL-MFE, it changes to 400 and 512 for LeNet-5 and
VGG-16.

###  V-B Effect of non-IID

Fig. 1 shows that as the degree of non-IID increases ($\alpha$ get smaller),
the local test accuracy of the Local also increases. It consistent with our
intuition: the more skewed the client’s data distribution, the easier it is to
get high local test accuracy (e.g., in a client, a few categories account for
a large proportion while only a small amount of data belongs to other
categories, or the client has only one category at all). However, The model in
Local is not necessarily a good model because the categories with only a small
amount of data may be discarded, so that it never really distinguishes the
difference between each category. The conjecture is confirmed from the global
test of the Local in Fig. 1 and Tab. II., in all experiment, the global test
accuracy of all clients is particularly low. The highly non-IID skewed local
data leads to the limitation of the stand-alone trained model.

From Tab. I and Tab. II, in all experiments, the accuracy of Local is lower
than FedAvg’s in both local test and global test, whether it is a large model
VGG-16 or a small model LeNet-5. In the CIFAR-10 experiments of Local, the
accuracy of the larger model VGG-16 is particularly low, which is
significantly lower than the smaller model LeNet-5. Generally speaking, the
deeper the model, the higher the limitation of its capabilities, but more data
is required for training. Although the simple model performs relatively well
in Local, its limitation gives the client a stronger incentive to participate
in federated learning to train a more powerful model. Fig. 2 shows that VGG-16
achieves nearly 90% test accuracy by FedAvg algorithm. Large model can be
adequately trained through federated learning, which greatly improves model
accuracy compared to Local.

The column FedAvg in Tab. I (1000th round in Fig. 2) presents the test
accuracy of the global model of the FedAvg, where all three groups of
experiments have the same results: the smaller the $\alpha$, the worse the
performance of FedAvg. On the contrary, Tab. I shows that as $\alpha$ gets
smaller, the potential for personalization is growing. In DML, the convergence
relies on the assumption that data are IID among workers. In line with the
theoretical analysis [20], a smaller $\alpha$ means more deviation from the
IID assumption, which hurts convergence rate and final model performance of
FedAvg. But this also makes us think: Why use the same global model for
clients with heterogenous data. We further show that federated personalized
learning is naturally suitable for the non-IID data distribution, and can
effectively improve overall performance.

Figure 3: Fashion-MNIST + LeNet-5, $\alpha=0.9$, 100 clients. The global test
accuracy and local test accuracy of all client of PFL-FB, PFL-MF, and PFL-MFE
algorithms. All x-axis are FedAvg local test accuracy of each client (can be
regarded as client index). Each point represents a test accuracy comparison
between a PFL algorithm and FedAvg for a particular client. Compared with PFL-
FB baseline, the more red triangles, the better our algorithms. Fig. 3$(a)$
and Fig. 3$(b)$ compare the local test accuracy of the four algorithms for
each client, while Fig. 3$(c)$ and Fig 3$(d)$ compare the global test accuracy
of them for each client. Fig.4 and Fig.4 have the same meaning but with
different dataset+model.

###  V-C Personalization of PFL-FB and PFL-MF

Tab. II shows that the average global test accuracy of these three
personalization algorithms are higher than that of Local, which means that
personalized federated learning can effectively improve the generalization
ability of the model on client. Tab. 1 shows that the average local test
accuracy of these three personalization algorithms are 2% $\sim$ 10% higher
than that of FedAvg at $\alpha=0.5$. However, PFL-FB is more likely to cause
overfitting. When $\alpha$ is relatively large, PFL-FB is more likely to
overfit local data, which makes the personalized model worse than the global
model. In the proposed algorithm PFL-MF, if the personalized model
deteriorates, it will be directly discarded by the gating network. Moreover,
the gating network has learned which data is appropriate to use the
personalized model and which data needs to be weighted more toward the global
model. Consequently, PFL-MF is more stable, and its local test accuracy is
always higher than FedAvg.

Comparing PFL-FB with PFL-MF in Tab. I and Tab. II, for each client, the
increase of the local test accuracy of PFL-FB comes at the cost of a
significant drop in global test accuracy. For example, in the CIFAR-10 +
LeNet-5 experiment, the average global test accuracy of PFL-FB dropped by 15%,
from 68.92% to 54.28%. PFL-MF achieves the average global test accuracy with
1% $\sim$ 8% higher than that of PFL-FB in all experiments. The main incentive
for clients participating in federated learning is to obtain a better model
that is both personalized and generalized. However, for a single model, the
two capabilities are often contradictory. The algorithm based on our PFL-MoE
makes it possible to get both capabilities at the same time.

As shown in Tab. I and Tab. II, in experiments on Fashion-MNIST, the proposed
PFL-MF algorithm performs better than PFL-FB on the two tests, which indicates
that PFL-MF achieves better personalization and better generalization than
PFL-FB. However, in experiments on CIFAR-10, the average local test accuracy
of PFL-MF is lower than that of PFL-FB, except in the LeNet-5 experiment with
$\alpha=2$. It can be observed more obviously from Fig. 3$(a)$, Fig. 4$(a)$,
and Fig. 5$(a)$. In Fig. 3$(a)$, red is the majority, while in Fig. 4$(a)$ and
Fig. 5$(a)$, green is the majority. In PFL-FM, the gating network can only
recognize and make the decisions for simple data, such as the Fashion-MNIST
dataset. But for the more complex color image dataset CIFAR-10, the gating
network needs to be improved. ‘

###  V-D Effect of PFL-MFE

From Tab. I, in the experiments on CIFAR-10, the average local test of PFL-MFE
is 0.3% $\sim$ 1.7% higher than that of PFL-MF. Although the average global
test accuracy of PFL-MFE is slightly lower than that of PFL-MF, it is still
higher than that of PFL-FB. Comparing Fig. 4$(a)$ with Fig. 4$(b)$, we can see
that, lots of green triangles in Fig. 4$(a)$ increase and turn to red in Fig.
4$(b)$ while the triangles in Fig. 4$(c)$ and Fig. 4$(d)$ are always red. In
Fig. 5, we have the same observation as in Fig. 4. In the experiments on
Fashion-MNIST, although PFL-MFE outperforms PFL-MF in local test only when
$\alpha=0.5$, it is very close to PFL-MF at other $\alpha$ values. PFL-MFE and
PFL-MF are both 0.05% $\sim$ 0.5% higher than PFL-FB in average local test
accuracy.

In the experiments on CIFAR-10, PFL-MFE is more effective than PFL-MF, because
the gating network can make better decisions indirectly by using the abstract
features. For the simple Fashion-MNIST dataset, PFL-MF is enough. PFL-MFE can
better recognize complex input data while maintaining the advantages of PFL-
MF. As shown in Tab. I and Tab. II, in all experiments, overall, PFL-MFE
achieves the best average local test accuracy among three personalization
algorithms, and it is much better than PFL-FB in the average global test
accuracy. In a word, the proposed PFL-MFE algorithm can realize
personalization more effectively while maintaining better generalization.

##  VI Conclusion and Future work

We propose PFL-MoE, an MoE based approach for personalized federated learning.
PFL-MoE can combine advantages of the personalized model and the global model
and achieve both better personalization and better generalization. PFL-MF, an
instance of PFL-MoE, performs better than existing algorithm PFL-FB in both
local test and global test. We also show that PFL-MFE strengthens the
decision-making ability of the MoE gating network and can be effective when
processing more complex data. In the future, we will explore the instance of
PFL-MoE of other personalized federated learning algorithms and compare their
overall performance.

Figure 4: CIFAR-10 + LeNet-5, $\alpha=0.9$, 100 clients. Comparing Fig. 4$(a)$
with Fig. 4$(b)$, the same in Fig. 3, the points with $y>0$ in the triangle
points are more than that in round points. But in Fig. 4$(a)$, there are a
large number of green triangles which means that the personalization effect of
PFL-MF is not better as that of PFL-FB. But from Fig. 4$(b)$, the extension
algorithm FL-MFE works better than PFL-FB.

Figure 5: CIFAR-10 + VGG-16, $\alpha=2.0$, 100 clients. Just like Fig. 4, PFL-
MFE shows its advantage on the CIFAR-10 dataset. Comparing Fig. 5$(c)$ with
Fig. 5$(d)$, almost all triangles are red, which is also shown in Fig. 3 and
Fig. 4, indicating that our algorithms are always superior to PFL-FB in the
global test.

## References

  * [1] T. Ben-Nun and T. Hoefler, “Demystifying parallel and distributed deep learning: An in-depth concurrency analysis,” _ACM Computing Surveys (CSUR)_ , vol. 52, no. 4, pp. 1–43, 2019.
  * [2] X. Wang, Y. Han, C. Wang, Q. Zhao, and M. Chen, “In-edge ai: Intelligentizing mobile edge computing, caching and communication by federated learning,” _IEEE Network_ , vol. PP, no. 99, pp. 1–10, 2019.
  * [3] B. McMahan, D. Ramage, K. Talwar, and L. Zhang, “Learning differentially private recurrent language models,” in _International Conference on Learning Representations (ICLR)_ , 2018. [Online]. Available: https://openreview.net/pdf?id=BJ0hF1Z0b
  * [4] T. Yu, E. Bagdasaryan, and V. Shmatikov, “Salvaging federated learning by local adaptation,” _arXiv preprint arXiv:2002.04758_ , 2020.
  * [5] K. Wang, R. Mathews, C. Kiddon, H. Eichner, F. Beaufays, and D. Ramage, “Federated evaluation of on-device personalization,” _arXiv preprint arXiv:1910.10252_ , 2019.
  * [6] V. Smith, C.-K. Chiang, M. Sanjabi, and A. S. Talwalkar, “Federated multi-task learning,” in _Advances in Neural Information Processing Systems_ , 2017, pp. 4424–4434.
  * [7] D. Li and J. Wang, “Fedmd: Heterogenous federated learning via model distillation,” _arXiv preprint arXiv:1910.03581_ , 2019.
  * [8] D. Peterson, P. Kanani, and V. J. Marathe, “Private federated learning with domain adaptation,” _arXiv preprint arXiv:1912.06733_ , 2019.
  * [9] R. A. Jacobs, M. I. Jordan, S. J. Nowlan, and G. E. Hinton, “Adaptive mixtures of local experts,” _Neural computation_ , vol. 3, no. 1, pp. 79–87, 1991\.
  * [10] S. Masoudnia and R. Ebrahimpour, “Mixture of experts: a literature survey,” _Artificial Intelligence Review_ , vol. 42, no. 2, pp. 275–293, 2014.
  * [11] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas, “Communication-efficient learning of deep networks from decentralized data,” in _Artificial Intelligence and Statistics_. PMLR, 2017, pp. 1273–1282.
  * [12] J. Konečný, H. B. McMahan, F. X. Yu, P. Richtarik, A. T. Suresh, and D. Bacon, “Federated learning: Strategies for improving communication efficiency,” in _NIPS Workshop on Private Multi-Party Machine Learning_ , 2016. [Online]. Available: https://arxiv.org/abs/1610.05492
  * [13] T. Li, A. K. Sahu, A. Talwalkar, and V. Smith, “Federated learning: Challenges, methods, and future directions,” _IEEE Signal Processing Magazine_ , vol. 37, no. 3, pp. 50–60, 2020.
  * [14] P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings _et al._ , “Advances and open problems in federated learning,” _arXiv preprint arXiv:1912.04977_ , 2019.
  * [15] L. Melis, C. Song, E. De Cristofaro, and V. Shmatikov, “Exploiting unintended feature leakage in collaborative learning,” in _2019 IEEE Symposium on Security and Privacy (SP)_. IEEE, 2019, pp. 691–706.
  * [16] L. Lyu, H. Yu, and Q. Yang, “Threats to federated learning: A survey,” _arXiv preprint arXiv:2003.02133_ , 2020.
  * [17] T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith, “Federated optimization in heterogeneous networks,” in _Proceedings of Machine Learning and Systems 2020, MLSys 2020, Austin, TX, USA, March 2-4, 2020_ , I. S. Dhillon, D. S. Papailiopoulos, and V. Sze, Eds. mlsys.org, 2020. [Online]. Available: https://proceedings.mlsys.org/book/316.pdf
  * [18] H. Hsu, H. Qi, and M. Brown, “Measuring the effects of non-identical data distribution for federated visual classification,” 2019. [Online]. Available: https://arxiv.org/abs/1909.06335
  * [19] S. Caldas, S. M. K. Duddu, P. Wu, T. Li, J. Konečnỳ, H. B. McMahan, V. Smith, and A. Talwalkar, “Leaf: A benchmark for federated settings,” _arXiv preprint arXiv:1812.01097_ , 2018.
  * [20] X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang, “On the convergence of fedavg on non-iid data,” in _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020. [Online]. Available: https://openreview.net/forum?id=HJxNAnVtDS
  * [21] Y. Zhao, M. Li, L. Lai, N. Suda, D. Civin, and V. Chandra, “Federated learning with non-iid data,” _arXiv preprint arXiv:1806.00582_ , 2018.
  * [22] V. Kulkarni, M. Kulkarni, and A. Pant, “Survey of personalization techniques for federated learning,” _arXiv preprint arXiv:2003.08673_ , 2020.
  * [23] Y. Jiang, J. Konečnỳ, K. Rush, and S. Kannan, “Improving federated learning personalization via model agnostic meta learning,” _In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1126–1135. JMLR._ , 2017.
  * [24] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_ , ser. Proceedings of Machine Learning Research, D. Precup and Y. W. Teh, Eds., vol. 70. PMLR, 2017, pp. 1126–1135. [Online]. Available: http://proceedings.mlr.press/v70/finn17a.html
  * [25] C. T Dinh, N. Tran, and T. D. Nguyen, “Personalized federated learning with moreau envelopes,” _Advances in Neural Information Processing Systems_ , vol. 33, 2020.
  * [26] L. Corinzia and J. M. Buhmann, “Variational federated multi-task learning,” _arXiv preprint arXiv:1906.06268_ , 2019.
  * [27] F. Hanzely and P. Richtárik, “Federated learning of a mixture of global and local models,” _arXiv preprint arXiv:2002.05516_ , 2020.
  * [28] Y. Deng, M. M. Kamani, and M. Mahdavi, “Adaptive personalized federated learning,” _arXiv preprint arXiv:2003.13461_ , 2020.
  * [29] Y. Mansour, M. Mohri, J. Ro, and A. T. Suresh, “Three approaches for personalization with applications to federated learning,” _arXiv preprint arXiv:2002.10619_ , 2020.
  * [30] H. Xiao, K. Rasul, and R. Vollgraf. (2017) Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.
  * [31] A. Krizhevsky, G. Hinton _et al._ , “Learning multiple layers of features from tiny images,” 2009.
  * [32] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” _Proceedings of the IEEE_ , vol. 86, no. 11, pp. 2278–2324, 1998.
  * [33] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _Computer ence_ , 2014.


# This is the title

LOCALIZED BOUNDED BELOW APPROXIMATE SCHAUDER FRAMES ARE FINITE UNIONS OF
APPROXIMATE RIESZ SEQUENCES
K. MAHESH KRISHNA
Post Doctoral Fellow

Statistics and Mathematics Unit

Indian Statistical Institute, Bangalore Centre

Karnataka 560 059, India

Email<EMAIL_ADDRESS>
Date:

Abstract: Based on the truth of Feichtinger conjecture by Marcus, Spielman and
Srivastava [Ann. of Math. (2), 2015] and from the localized version by
Gröchenig [Adv. Comput. Math., 2003], we introduce the notion of localization
of approximate Schauder frames (ASFs) and approximate Riesz sequences (ARSs).
We show that localized bounded below ASFs are finite unions of ARSs.

Keywords: Feichtinger conjecture, Frame, Riesz sequence, Localization.

Mathematics Subject Classification (2020): 42C15, 46A45, 46B45.

##  1\. Introduction

In the beginning years of $21^{th}$ century, Prof. Feichtinger formulated the
following conjecture based on his extensive work on Gabor/Weyl-Heisenberg
frames (see [13] for the history and [36, 16, 14, 19, 18, 28, 17, 25, 26, 4,
22, 27] for general theory).

######  Conjecture 1.1.

[8] (Feichtinger Conjecture/Marcus-Spielman-Srivastava Theorem) Let
$\\{\tau_{n}\\}_{n}$ be a frame for a Hilbert space $\mathcal{H}$ such that

$\displaystyle 0<\inf_{n\in\mathbb{N}}\|\tau_{n}\|.$

Then $\\{\tau_{n}\\}_{n}$ can be partitioned into a finite union of Riesz
sequences for $\mathcal{H}$.

First breakthrough which supported Conjecture 1.1 occurred when Gröchenig
proved it for intrinsically localized frames [23]. Shortly afterwords, it has
been verified for certain classes of $\ell^{1}$-self-localized frames, wavelet
frames, Gabor frames, frames of translates, frames formed by reproducing
kernels and exponential frames/frames of exponentials [6, 2, 35, 29, 3, 30,
39]. Conjecture 1.1 received great attention after establishing its
equivalence with Kadison-Singer conjecture [10, 9, 12]. Finally, the
Feichtinger conjecture has been solved fully by resolving Weaver’s conjecture
by Marcus, Spielman, and Srivastava in 2013 [34, 38, 5, 37, 33]. In this
paper, we formulate a Banach space version of Conjecture 1.1 and prove it for
bounded below intrinsically localized ASFs (Theorem 2.7).

##  2\. Localized bounded below ASFs are finite unions of ARSs

We consider the following most general notion of approximate Schauder frames.
In the entire paper, $\mathcal{X}$ is a separable Banach space and
$\mathcal{X}^{*}$ is its dual.

######  Definition 2.1.

[11, 7, 21] Let $\\{\tau_{n}\\}_{n}$ be a collection in $\mathcal{X}$ and
$\\{f_{n}\\}_{n}$ be a collection in $\mathcal{X}^{*}$. The pair
$(\\{f_{n}\\}_{n},\\{\tau_{n}\\}_{n})$ is said to be an approximate Schauder
frame (we write ASF) for $\mathcal{X}$ if the frame operator

$\displaystyle S_{f,\tau}:\mathcal{X}\ni x\mapsto
S_{f,\tau}x\coloneqq\sum_{n=1}^{\infty}f_{n}(x)\tau_{n}\in\mathcal{X}$

is a well-defined bounded linear invertible operator.

We use the following notion of ‘bounded below’ for ASFs.

######  Definition 2.2.

An ASF $(\\{f_{n}\\}_{n},\\{\tau_{n}\\}_{n})$ for $\mathcal{X}$ is said to be
bounded below if

$\displaystyle\inf_{n\in\mathbb{N}}|f_{n}(\tau_{n})|>0.$

Motivated from the definition of localization of frames [24, 20], we introduce
the following notion.

######  Definition 2.3.

An ASF $(\\{f_{n}\\}_{n},\\{\tau_{n}\\}_{n})$ for $\mathcal{X}$ is said to be
intrinsically/self localized if there exist $s>1$ and $A>0$ such that

$\displaystyle|f_{n}(\tau_{m})|\leq\frac{A}{(1+|n-m|)^{s}},\quad\forall
n,m\in\mathbb{N}.$

Two notions of Riesz sequences for Banach spaces exist in literature, see [15,
1] and [32]. Here we define another.

######  Definition 2.4.

An ASF $(\\{f_{n}\\}_{n},\\{\tau_{n}\\}_{n})$ for $\mathcal{X}$ is said to be
an approximate Riesz sequence (we write ARS) if there exists a finite
partition $Q_{1},\dots,Q_{N}$ of $\mathbb{N}$ such that

$\displaystyle\mathbb{N}=\bigcup_{j=1}^{N}Q_{j}$

and for each $1\leq j\leq N$,

$\displaystyle\inf_{n\in Q_{j}}\left(|f_{n}(\tau_{n})|-\sum_{m\in Q_{j},m\neq
n}|f_{n}(\tau_{m})|\right)>0.$

Note that for Hilbert spaces, if $\\{f_{n}\\}_{n}$ is determined by
$\\{\tau_{n}\\}_{n}$ (Riesz representation), then Definition 2.4 is equivalent
(due to positivity) to the definition of Riesz sequence (see[23]). We now
formulate the following conjecture (some other are formulated in [31]).

######  Conjecture 2.5.

Every bounded below ASF can be partitioned as a finite union of ARBs.

We now prove Conjecture 2.5 for intrinsically localized ASFs with the help of
following result.

######  Theorem 2.6.

[23]

  1. (i)

For every $s>1$,

$\displaystyle
D_{s}\coloneqq\sup_{x\in\mathbb{R}}\sum_{n=1}^{\infty}\frac{1}{(1+|n-x|)^{s}}<\infty.$

  2. (ii)

For every $s>1$, there exists a $C_{s}>0$ (which does not depend on $\delta$)
such that

$\displaystyle\sup_{m\in\mathbb{N}}\sum_{n\in\mathbb{N},n\neq
m}\frac{1}{(1+|n-m|)^{s}}\leq\frac{C_{s}}{\delta^{s}},$

whenever

$\displaystyle\inf_{n,m\in\mathbb{N},n\neq m}|n-m|\geq\delta.$

######  Theorem 2.7.

Conjecture 2.5 holds for intrinsically localized bounded below ASFs.

###### Proof.

Our proof is highly motivated from [23]. Let
$(\\{f_{n}\\}_{n},\\{\tau_{n}\\}_{n})$ be an intrinsically localized bounded
below ASF for $\mathcal{X}$. Define

$\displaystyle C\coloneqq\inf_{n\in\mathbb{N}}|f_{n}(\tau_{n})|>0.$

Since $(\\{f_{n}\\}_{n},\\{\tau_{n}\\}_{n})$ intrinsically localized there
exist $s>1$ and $A>0$ such that

$\displaystyle|f_{n}(\tau_{m})|\leq\frac{A}{(1+|n-m|)^{s}},\quad\forall
n,m\in\mathbb{N}.$

Let $C_{s}$ be the constant as in Theorem 2.6. Choose natural number $M$ such
that

$\displaystyle\frac{AC_{s}}{M^{s}}\leq\frac{C}{2}.$

We now partition $\mathbb{N}$ into $Q_{1},\dots,Q_{N}$ such that

$\displaystyle\mathbb{N}\coloneqq\bigcup_{j=1}^{N}Q_{j}$

and for each $1\leq j\leq N$,

$\displaystyle\inf_{n,m\in Q_{j},n\neq m}|n-m|\geq M.$

(Note that there are infinitely many partitions of $\mathbb{N}$ of this type.)
Let $1\leq j\leq N$. Then using (ii) in Theorem 2.6

$\displaystyle\sup_{n\in Q_{j}}\sum_{m\in Q_{j},m\neq n}|f_{n}(\tau_{m})|\leq
A\sup_{n\in Q_{j}}\sum_{m\in Q_{j},m\neq n}\frac{1}{(1+|n-m|)^{s}}\leq
A\frac{C_{s}}{M^{s}}\leq\frac{C}{2}.$

Therefore for each fixed $1\leq j\leq N$

$\displaystyle\inf_{n\in Q_{j}}\left(|f_{n}(\tau_{n})|-\sum_{m\in Q_{j},m\neq
n}|f_{n}(\tau_{m})|\right)$ $\displaystyle=\inf_{n\in
Q_{j}}|f_{n}(\tau_{n})|-\sup_{n\in Q_{j}}\sum_{m\in Q_{j},m\neq
n}|f_{n}(\tau_{m})|$ $\displaystyle\geq C-\frac{C}{2}=\frac{C}{2}>0.$

Since $j$ was arbitrary, we get the theorem. ∎

## References

  * [1] Akram Aldroubi, Qiyu Sun, and Wai-Shing Tang.  $p$-frames and shift invariant subspaces of $L^{p}$.  J. Fourier Anal. Appl., 7(1):1–21, 2001.
  * [2] Radu Balan, Peter G. Casazza, Christopher Heil, and Zeph Landau.  Density, overcompleteness, and localization of frames. I. Theory.  J. Fourier Anal. Appl., 12(2):105–143, 2006.
  * [3] Anton Baranov and Konstantin Dyakonov.  The Feichtinger conjecture for reproducing kernels in model subspaces.  J. Geom. Anal., 21(2):276–287, 2011.
  * [4] John J. Benedetto and David F. Walnut.  Gabor frames for $L^{2}$ and related spaces.  In Wavelets: mathematics and applications, Stud. Adv. Math., pages 97–162. CRC, Boca Raton, FL, 1994.
  * [5] Marcin Bownik.  The Kadison-Singer problem.  In Frames and harmonic analysis, volume 706 of Contemp. Math., pages 63–92. Amer. Math. Soc., [Providence], RI, 2018.
  * [6] Marcin Bownik and Darrin Speegle.  The Feichtinger conjecture for wavelet frames, Gabor frames and frames of translates.  Canad. J. Math., 58(6):1121–1143, 2006.
  * [7] P. G. Casazza, S. J. Dilworth, E. Odell, Th. Schlumprecht, and A. Zsák.  Coefficient quantization for frames in Banach spaces.  J. Math. Anal. Appl., 348(1):66–86, 2008.
  * [8] Peter G. Casazza, Ole Christensen, Alexander M. Lindner, and Roman Vershynin.  Frames and the Feichtinger conjecture.  Proc. Amer. Math. Soc., 133(4):1025–1033, 2005.
  * [9] Peter G. Casazza and Dan Edidin.  Equivalents of the Kadison-Singer problem.  In Function spaces, volume 435 of Contemp. Math., pages 123–142. Amer. Math. Soc., Providence, RI, 2007.
  * [10] Peter G. Casazza, Matthew Fickus, Janet C. Tremain, and Eric Weber.  The Kadison-Singer problem in mathematics and engineering: a detailed account.  In Operator theory, operator algebras, and applications, volume 414 of Contemp. Math., pages 299–355. Amer. Math. Soc., Providence, RI, 2006.
  * [11] Peter G. Casazza, Deguang Han, and David R. Larson.  Frames for Banach spaces.  In The functional and harmonic analysis of wavelets and frames (San Antonio, TX, 1999), volume 247 of Contemp. Math., pages 149–182. Amer. Math. Soc., Providence, RI, 1999.
  * [12] Peter G. Casazza, Gitta Kutyniok, Darrin Speegle, and Janet C. Tremain.  A decomposition theorem for frames and the Feichtinger conjecture.  Proc. Amer. Math. Soc., 136(6):2043–2053, 2008.
  * [13] Ole Christensen.  Six (seven) problems in frame theory.  In New perspectives on approximation and sampling theory, Appl. Numer. Harmon. Anal., pages 337–358. Birkhäuser/Springer, Cham, 2014.
  * [14] Ole Christensen.  An introduction to frames and Riesz bases.  Applied and Numerical Harmonic Analysis. Birkhäuser/Springer, second edition, 2016.
  * [15] Ole Christensen and Diana T. Stoeva.  $p$-frames in separable Banach spaces.  Adv. Comput. Math., 18(2-4):117–126, 2003.
  * [16] R. J. Duffin and A. C. Schaeffer.  A class of nonharmonic Fourier series.  Trans. Amer. Math. Soc., 72:341–366, 1952.
  * [17] Hans G. Feichtinger and K. Gröchenig.  Gabor frames and time-frequency analysis of distributions.  J. Funct. Anal., 146(2):464–495, 1997.
  * [18] Hans G. Feichtinger and Thomas Strohmer, editors.  Gabor analysis and algorithms: Theory and application.  Applied and Numerical Harmonic Analysis. Birkhäuser Boston, Inc., Boston, MA, 1998.
  * [19] Hans G. Feichtinger and Thomas Strohmer, editors.  Advances in Gabor analysis.  Applied and Numerical Harmonic Analysis. Birkhäuser Boston, Inc., Boston, MA, 2003.
  * [20] Massimo Fornasier and Karlheinz Gröchenig.  Intrinsic localization of frames.  Constr. Approx., 22(3):395–415, 2005.
  * [21] D. Freeman, E. Odell, Th. Schlumprecht, and A. Zsák.  Unconditional structures of translates for $L_{p}(\mathbb{R}^{d})$.  Israel J. Math., 203(1):189–209, 2014.
  * [22] Karlheinz Gröchenig.  Foundations of time-frequency analysis.  Applied and Numerical Harmonic Analysis. Birkhäuser Boston, Inc., Boston, MA, 2001.
  * [23] Karlheinz Gröchenig.  Localized frames are finite unions of Riesz sequences.  Adv. Comput. Math., 18(2-4):149–157, 2003.
  * [24] Karlheinz Gröchenig.  Localization of frames, Banach frames, and the invertibility of the frame operator.  J. Fourier Anal. Appl., 10(2):105–132, 2004.
  * [25] Karlheinz Gröchenig.  Gabor frames without inequalities.  Int. Math. Res. Not. IMRN, (23):Art. ID rnm111, 21, 2007.
  * [26] Karlheinz Gröchenig.  The mystery of Gabor frames.  J. Fourier Anal. Appl., 20(4):865–895, 2014.
  * [27] Karlheinz Gröchenig, Joaquim Ortega-Cerdà, and José Luis Romero.  Deformation of Gabor systems.  Adv. Math., 277:388–425, 2015.
  * [28] Christopher Heil.  History and evolution of the density theorem for Gabor frames.  J. Fourier Anal. Appl., 13(2):113–166, 2007.
  * [29] Sneh Lata and Vern Paulsen.  The Feichtinger conjecture and reproducing kernel Hilbert spaces.  Indiana Univ. Math. J., 60(4):1303–1317, 2011.
  * [30] Wayne Lawton.  The Feichtinger conjecture for exponentials.  J. Nonlinear Anal. Optim., 2(1):131–140, 2011.
  * [31] K. Mahesh Krishna.  Feichtinger conjectures, ${R}_{\varepsilon}$-conjectures and Weaver’s conjectures for Banach spaces.  arXiv.org/2201.00125v1 [math.FA] 1 January, 2022.
  * [32] K. Mahesh Krishna and P. Sam Johnson.  Dilation theorem for p-approximate Schauder frames for separable Banach spaces.  Palest. J. Math., 11(2):384–394, 2022.
  * [33] Adam Marcus and Nikhil Srivastava.  The solution of the Kadison-Singer problem.  In Current developments in mathematics 2016, pages 111–143. Int. Press, Somerville, MA, 2018.
  * [34] Adam W. Marcus, Daniel A. Spielman, and Nikhil Srivastava.  Interlacing families II: Mixed characteristic polynomials and the Kadison-Singer problem.  Ann. of Math. (2), 182(1):327–350, 2015.
  * [35] Darrin Speegle.  Uniform partitions of frames of exponentials into Riesz sequences.  J. Math. Anal. Appl., 348(2):739–745, 2008.
  * [36] Diana T. Stoeva.  On a characterization of Riesz bases via biorthogonal sequences.  J. Fourier Anal. Appl., 26(4):Paper No. 67, 5, 2020.
  * [37] Dan Timotin.  The solution of the Kadison-Singer problem.  In Recent advances in operator theory and operator algebras, pages 117–148. CRC Press, Boca Raton, FL, 2018.
  * [38] Nik Weaver.  The Kadison-Singer problem in discrepancy theory.  Discrete Math., 278(1-3):227–239, 2004.
  * [39] Eric Weber.  Algebraic aspects of the paving and Feichtinger conjectures.  In Topics in operator theory. Volume 1. Operators, matrices and analytic functions, volume 202 of Oper. Theory Adv. Appl., pages 569–578. Birkhäuser Verlag, Basel, 2010.


# The spread of COVID-19 at Hot-Temperature Places With Different Curfew
Situations Using Copula Models

Fadhah Alanazi  Deanship of Educational Services,
Department of General Sciences,
Prince Sultan University,
Riyadh, Saudi Arabia
<EMAIL_ADDRESS>
## Abstract

The infectious coronavirus disease 2019 (COVID-19) has become a serious global
pandemic. Different studies have shown that increasing temperature can play a
crucial role in the spread of the virus. Most of these studies were limited to
winter or moderate temperature levels and were conducted using conventional
models. However, traditional models are too simplistic to investigate complex,
non-linear relationships and suffer from some restrictions. Therefore, we
employed copula models to examine the impact of high temperatures on virus
transmission. The findings from the copula models showed that there was a weak
to moderate effect of temperature on the number of infections and the effect
almost vanished under a lockdown policy. Therefore, this study provides new
insight into the relationship between COVID-19 and temperature, both with and
without social isolation practices. Such results can lead to improvements in
our understanding of this new virus. In particular, the results derived from
the copula models examined here, unlike existing traditional models, provide
evidence that there is no substantial influence of high temperatures on the
active COVID-19 outbreak situation. In addition, the results indicate that the
transmission of COVID-19 is strongly influenced by social isolation practices.
To the best of the authors’ knowledge, this is the first copula model
investigation applied to the COVID-19 pandemic.

## Introduction

In December 2019, a novel infectious disease termed coronavirus disease 2019
(COVID-19) was discovered in Wuhan city, Hubei province, China. Subsequently,
and through human-to human transmission, this virus has caused a global
pandemic. COVID-19 is characterized by clinical features similar to those
caused by severe acute respiratory syndrome coronavirus (SARS-CoV) and Middle
Eastern respiratory syndrome coronavirus (MERS-CoV) infections, such as a
fever and dry cough [1].

Previous studies have shown that meteorological variables can affect the
transmission and survival of coronaviruses [2, 3]. Earlier research [3] found
that MERS-CoV is most active at high temperatures and low humidity.

Notably, recent studies have shown that warm weather and high humidity may be
important factors for reducing the spread of COVID-19 (e.g., see [4]).
Conversely, some existing studies have found that increasing temperatures will
not affect the transmission of COVID-19 (e.g., see [5]). However, most of
these studies were limited to winter or low-temperature weather with a small
number of observations. Hence, there is still no definitive evidence as to
whether there is a negative association between environmental variables and
the spread of COVID-19 in extremely hot or cold locations [4]. Besides, most
previous studies were performed using traditional models, which are too
simplistic and may be unable to deal with complex, non-linear dependency
patterns. Thus, further research to understand the activity of COVID-19 under
high-temperature conditions is warranted. In addition, such an association
should be investigated not only in regard to weather variables, but also by
taking into account the lockdown situation at these locations. Presently,
copula models have become a favored statistical tool to describe the
association between variables. These models have been applied in different
areas, including the study of infectious diseases (e.g., see [6]) and
environmental science (e.g., see [7]). One important benefit of using a copula
model is that one can model the marginal distribution independently from the
dependency structures, which are completely captured via the copula function.
Another benefit of using a copula model is that the margins do not need to
follow the same parametric family. Furthermore, many copula families exist,
each with its own capability to describe the unique dependency structure.
Hence, various types of associations can be discovered via copula models.

Hence, this study aimed to perform flexible statistical modeling with a copula
model to improve our knowledge about the spread of the virus in hot locations
with different curfew levels. Specifically, we investigated the impact of high
temperatures on the number of confirmed cases in the cities of Riyadh, Jeddah,
and Mecca in Saudi Arabia, and these cities were selected for several reasons.
First, Saudi Arabia has been strongly affected by MERS-CoV [8, 9], which
produces a similar severe respiratory illness as COVID-19. Second, the highest
numbers of confirmed cases in Saudi Arabia have been recorded in Riyadh,
Jeddah, and Mecca, which are three of the hottest areas in Saudi Arabia.
Third, because of the transmission of COVID-19, Mecca and Jeddah have been
placed under a series of lockdowns for a long time. Riyadh, however, was only
placed under a curfew for a short period. Hence, these cities represent strong
to moderate lockdown situations, which could be a factor critical to
understanding the effects of high temperatures on the spread of COVID-19. By
using the data from these cities and capitalizing on the flexibility of copula
models, we aimed to provide clear evidence on the association between high
temperatures and confirmed cases of COVID-19.

## Materials and methods

### Data collection

For this study, the cities of Riyadh, Mecca, and Jeddah were selected for the
analysis. Riyadh is the capital city of Saudi Arabia and the city most
affected by COVID-19 in this country. Riyadh had $37,244$ confirmed cases for
the observed period from $13$ March $2020$ to $15$ June $2020$. The population
of Riyadh estimated for the middle of 2018 based on demographic survey data
collected in 2016 was $8,446,866$ [10]. Jedda and Mecca are the second most
affected cities in Saudi Arabia, with $21,152$ and $20,248$ confirmed cases,
respectively.
Daily counts of confirmed cases for the study period were collected from
official reports [11] based on information [12] for Saudi Arabia. The daily
average temperature data for the same period time were obtained from the
Weather Underground Company [13].

### Data analysis

Both the COVID-19 confirmed cases and the average temperature data
demonstrated a non-normal distribution for all cities. Fig 1 1 shows the
confirmed cases during the study period, where the new confirmed cases of
COVID-19 in Riyadh exceeded $1500$ from $08$ to $15$ June $2020$. However, the
highest records for Mecca and Jeddah were generally similar and lower than
those of Riyadh.

Figure 1: Plots of the confirmed cases during the study period (from $13$
March $2020$ to $15$ June $2020$. (top): Riyadh, (middle): Mecca, (bottom)
Jeddah.

### Copula

Copula is a Latin word that means joins or links. A copula function refers to
a multivariate function that joins the multivariate distribution functions to
their univariate standard uniform margins [14]. Formally, a copula can be
defined as follows: copulas [15] are multivariate cumulative distribution
functions with uniform marginal distributions on (0,1) such that:

$C:[0,1]^{n}\rightarrow[0,1],\ \ \ \ \ n\geq 2.$ (1)

Sklar’s theorem [16] is the key rule of the copula function, and it can be
introduced as follows:

######  Theorem .1 (Sklar’s theorem)

If $F$ is an $n$-variate distribution function with univariate margins
$F_{1},F_{2},.....,F_{n}$, then there exists an $n$-variate copula function,
$C$, such that $\forall$
$\mathbf{x}=(x_{1},..,x_{n})^{{}^{\prime}}\in\mathcal{R}^{n}$:

$F(x_{1},x_{2},....,x_{n})=C(F_{1}(x_{1}),F_{2}(x_{2}),....,F_{n}(x_{n})).$
(2)

If the margins are continuous, then the copula

$C(u_{1},u_{2},....,u_{n})=F(F^{-1}_{1}(u_{1}),F^{-1}_{2}(u_{2}),...,F^{-1}_{n}(u_{n}))$
(3)

is unique, where $F^{-1}$ is the inverse function of the margins and
$u\in[0,1]^{n}$. Conversely, if $F_{1},...,F_{n}$ are the marginal
distribution functions and $C$ is a copula function, then the function $F$
(defined by equation (2)) is a joint distribution function with margins
$F_{1},...,F_{n}$.

In accordance with Sklar’s theorem (2), a copula models the marginal
distributions separately from the dependency pattern, with no restriction on
the type of margins.

In this study, we consider an arbitrary number of copula types including the
Joe, Gumbel, and Clayton copulas, as well as their rotation types. In
addition, we consider the Frank, Gaussian, t-students, and other two-
parametric copulas, such as the Joe-Frank (BB8) copula. The following text
provides details on some commonly used copula families.

  * •

Frank copula is a one-parametric symmetric Archimedean copula with generator
function $\varphi(t)=-\ln[\frac{e^{-\theta t}-1}{e^{-\theta}-1}]$, with
$\theta\in(-\infty,\infty)$ $\backslash$ $\\{0\\}$. The Frank copula can
control both the negative and positive dependency pattern, where the strongest
dependency occurs at the center of the distribution. However, in the Frank
copula, the extremes are independent.

The distribution function of the Frank copula can be given by:

$C_{\theta}(u_{1},u_{2})=\frac{-1}{\theta}\ \ln\ [1+\frac{(e^{-\theta
u_{1}}-1)(e^{-\theta u_{2}}-1)}{e^{-\theta}-1}],$ (4)

and its density function is:

$c(u_{1},u_{2})=\theta(e^{-\theta}-1)\frac{e^{-\theta}(u_{1}+u_{2})}{e^{-\theta}-1+(e^{-\theta
u_{1}}-1)(e^{-\theta u_{2}}-1)}.$ (5)

  * •

Clayton copula is a one-parametric ($\theta>0$) non-symmetric Archimedean
copula. It is a lower positive tail dependence copula with generator
$\varphi(t)=\frac{1}{\theta}(t^{-\theta}-1)$. Its distribution is given by:

$C(u_{1},u_{2})=[u_{1}^{-\theta}+u_{2}^{-\theta}-1]^{\frac{-1}{\theta}},$ (6)

and its density function is:

$c(u_{1},u_{2})=(1+\theta)(u_{1}u_{2})^{-1-\theta}(u_{1}^{-\theta}+u_{2}^{-\theta}-1)^{\frac{-1}{\theta}-2}.$
(7)

  * •

Joe copula, in contrast to the Clayton copula, this is a one-parametric upper
tail Archimedean copula with generator $\varphi(t)=\ln[1-(1-t)^{\theta}]$. Its
distribution function is:

$C(u_{1},u_{2})=1-[(1-u_{1})^{\theta}+(1-u_{2})^{\theta}-(1-u_{1})^{\theta}(1-u_{2})^{\theta}]^{\frac{1}{\theta}},$
(8)

and its density function is:

$\begin{split}c(u_{1},u_{2})=[(1-u_{1})^{\theta}+(1-u_{2})^{\theta}-(1-u_{1})^{\theta}(1-u_{2})^{\theta}]^{\frac{1}{\theta}-2}\\\
\times(1-u_{1})^{\theta-1}(1-u_{2})^{\theta-1}[\theta-1+(1-u_{1})^{\theta}\\\
+(1-u_{2})^{\theta}-(1-u_{1})^{\theta}(1-u_{2})^{\theta}].\end{split}$ (9)

  * •

Rotated copula refers to a rotation version of asymmetric copulas. This
rotation includes $90$, $180$, and $270$ rotation degrees, with arguments
$(1-u_{1},u_{2}),(u_{1},1-u_{2})$, and $(1-u_{1},1-u_{2})$, respectively. The
$180$ rotation degree produces a corresponding survival copula family.
However, rotations by $90$ and $270$ degrees provide corresponding copulas to
deal with negative dependencies. For more details on rotated copulas, see for
example, [17], [18], [19], and [20].

#### Pseudo maximum-likelihood method

In this study, we applied the so-called pseudo maximum-likelihood method (PML)
to estimate the parameters for the selected copula function. PML is introduced
by [21] as a two-step estimation method. With this method, the margins are
estimated non-parametrically via their empirical cumulative distribution
function at first, and then, the copula parameter ($\theta_{c}$) is estimated
at the second step. By using PML, the copula parameter is estimated by
maximizing the copula density, i.e.,

$L_{\textbf{MPL}}(\theta_{c})=\sum_{i=1}^{n}\log[c(u_{1i},u_{2i};\theta_{c})],$
(10)

where $u_{1}=\hat{F}_{1}(x_{1};\alpha_{1})$ and
$u_{2}=\hat{F}_{2}(x_{2};\alpha_{2})$ are the empirical probability integral
transform of variable $X_{1}$ and $X_{2}$, respectively. A simulation study of
[22] showed that the performance of PML is better than that of the full
maximum likelihood estimation method and Inference Function of Margins of [23]
if the margins are unknown, which is the case in almost all real life
applications.

###  -A Goodness-of-fit test

As there is a wide range of copula functions, it is necessary to test the
copula shape with the best fit. Therefore, we will use the Akaike Information
Criterion (AIC) of [24] and the Bayesian Information Criterion (BIC) of [25]
to select the right copula. AIC and BIC can be given by:

$\texttt{AIC}=-2\ \texttt{ln}\ \texttt{L}(\hat{\theta})+2\\*\texttt{P},$ (11)
$\texttt{BIC}=-2\ \texttt{ln}\ \texttt{L}(\hat{\theta})+\texttt{P}\
(\texttt{ln (N))},$ (12)

where $\hat{\theta}$ is the estimated value of the parameters, and P is the
number of the model parameters.

The summary of the full inference steps of copula models used in this study is
as follows:

  * •

Transform the continuous variable of the observed data to copula data.

  * •

Calculate the cumulative density function for the discrete variable of the
observed data.

  * •

Consider arbitrary types of bivariate copula functions for the assumed model.

  * •

Select the best fit bivariate copula type among all fitted copula functions
using AIC and BIC.

## Results and discussion

### Descriptive results

Table I shows the summary statistics for the daily data on temperature and
COVID-19 confirmed cases in the cities of Riyadh, Mecca, and Jeddah. With
average values of $None$C (Riyadh), $None$C (Mecca), and $None$C (Jeddah), the
temperature in the three cities was very high. Importantly, this study takes
into account the daily average temperature, and not the maximum temperature.

TABLE I:  Summary statistics for the daily average temperature and COVID-19 confirmed cases in the cities of Riyadh, Mecca, and Jeddah. Zeros values indicate no new confirmed cases for the corresponding date during the observation period. Variable | N | Mean | Standard deviation | Min | Max
---|---|---|---|---|---
Temperature (Riyadh) (°C) | $95$ | $29.788$ | $4.947$ | $18$ | $37.4$
Temperature (Mecca) (°C) | 95 | 24.983 | 3.472 | 18 | 32
Temperature (Jeddah) (°C) | 95 | 29.014 | 3.258 | 22.9 | 36
Confirm cases (Riyadh) | 95 | 392.042 | 427.393 | 2 | 1,735
Confirm cases (Mecca) | 95 | 213.137 | 160.938 | 0 | 623
Confirm cases (Jeddah) | 95 | 222.653 | 174.632 | 0 | 586

### Discussion of the Copula model

This study used a copula model to investigate the relationship between high
temperatures and confirmed cases of COVID-19 for three cities in Saudi Arabia.
The existing number of copulas is large, and to select the most appropriate
fitted model for each city, bicop() function of R ([26]) package [27] was
used. As bicop() allows one to consider different selection criteria at each
run, it was applied to each data set twice, one with BIC and the other one
with AIC. The results of these models are provided in Tables (II, III).
Boldfont indicates the selected copula type. Figs (2, 3) present the surface
and contour plots for the selected copula families.

Figure 2: Surface plots of the Frank copulas for Riyadh (top) and Jeddah
(middle), and the Clayton copula for Mecca (Bottom) as the best-fit copula
families for each city.

Figure 3: Contour plots of the Frank copulas for Riyadh (top) and Jeddah (middle), and the Clayton copula for Mecca (Bottom) as the best-fit copula families for each city. TABLE II:  Selected copula families (for each city) based on BIC by bicop() Type | Parameter ($\tau$) | AIC | BIC
---|---|---|---
| Riyadh |  |
Frank |  $\theta=8.52$ ($\tau=0.62$) | $\mathbf{-102.5}$ | $\mathbf{-99.97}$
| Mecca |  |
Survival Joe (rotation degree $180$) |  $\theta=1.92$ ($\tau=0.43$) | $\mathbf{-35.07}$ | $\mathbf{-32.51}$
| Jeddah |  |
Frank |  $\theta=7.32$ ($\tau=0.57$) | $\mathbf{-85.52}$ | $\mathbf{-82.97}$
TABLE III: Selected copula families (for each city) based on AIC by bicop(). Type | Parameter ($\tau$) | AIC | BIC
---|---|---|---
| Riyadh |  |
BB8 (Joe-Frank) |  $\theta=8$ ($\alpha=0.7$) | $-103.74$ | $-98.63$
| Mecca |  |
tll 111(nonparametric (transformed kernel)) |  $7.47$ df 222(df = degree of freedom) | $-39.32$ | $-20.24$
| Jeddah |  |
Frank |  $\theta=7.32$ ($\tau=0.57$) | $\mathbf{-85.52}$ | $\mathbf{-82.97}$

In accordance with Table (II), Frank copulas with a moderate positive
dependency ($\theta=8.51$, $\tau=0.62$) and (($\theta=7.47$, $\tau=0.57$) were
selected for Riyadh and Jeddah, respectively. The results indicate that there
is a positive relationship between temperature and the spread of COVID-19 in
moderate and high temperatures. However, these two variables became
independent at extreme values. Therefore, the results provide clear evidence
that SARS-CoV-2 can still remain an active virus in hot places. In the case of
Mecca, the survival Joe copula with a low dependency level ($\theta=1.92$,
$\tau=0.34$) was selected as the most appropriate copula function. With these
data, there was only a very weak dependency pattern between high temperatures
and confirmed cases at the low values. This relationship reflects the period
before the series of lockdowns in the city of Mecca. During the curfew, there
was no relationship detected between the spread of COVID-19 and the
temperature.

The question that now remains is, do high temperatures affect the transmission
of COVID-19? To answer this question, we need to mention some of the main
similarities and differences among these cities regarding the (1) temperature,
(2) confirmed cases of COVID-19, (3) copula model results, and (4) lockdown
situations. First, the temperatures in Riyadh and Jeddah were almost the same
and slightly higher than those in Mecca. However, the number of confirmed
cases in Riyadh was higher than that in Jeddah and Mecca. In addition, Mecca
and Jeddah had almost the same number of confirmed cases. Hence, the same
temperature levels were associated with different numbers of new confirmed
cases within the three cities. Thus, COVID-19 can spread differently at the
same temperature level. Second, the fitted copula models show that the
dependency between the temperature and number of COVID-19 cases was very
similar for Riyadh and Jeddah, while it was very low for Mecca. Given the
first and second points mentioned above, there was another important factor
driving the active situation of this new virus, namely, the lockdown
situation. During the study period, the cities of Jeddah and Mecca were placed
under a lockdown on $29$ March $2020$ and $30$ March $2020$, respectively.
Then, Mecca was subjected to a $24$ hour curfew on $02$ April $2020$. Later,
on $04$ April $2020$, various areas in Jeddah were placed under a $24$ hour
curfew. After about one day, Riyadh too was placed under a $24$ hour curfew
for approximately $18$ days. Then, on $25$ April $2020$, Jeddah and Riyadh
were placed under partial curfews, while Mecca still remained under a $24$
lockdown. Hence, Mecca experienced a long lockdown period, while duration of
curfew in Riyadh was the shortest. These factors may explain the similarity in
copula results for Riyadh and Jeddah and the low dependency pattern in the
case of Mecca. In consideration of these last findings, we can conclude that
high temperatures had only a weak to moderate effect on the transmission of
COVID-19 if there was a partial curfew policy in place. However, this effect
vanished under the condition of strong social isolation. Hence, even in hot
places, COVID-19 can still spread readily when no social distancing is
implemented.

## Conclusion

This study examined the effect of high temperatures on the spread of COVID-19
in hot climates under different curfew situations using copula models. We
applied the models to the cities of Riyadh, Jeddah, and Mecca in Saudi Arabia.
For Riyadh and Jeddah, which had almost the same average temperature level,
the association between temperature and confirmed cases of COVID-19 reflected
a moderate positive Frank copula. However, the number of COVID-19 cases in
Riyadh was higher than the number in Jeddah. Hence, the transmission of this
virus in these two cities may have been affected by the curfew level and not
by the high temperature. In the case of Mecca, which had a temperature level
(slightly) less than that of Riyadh and Jeddah, there was a very weak
dependency between temperature and the number of COVID-19 cases. However, the
number of confirmed cases of COVID-19 in Mecca was very close to the number in
Jeddah. In addition, Mecca was under a strong $24$ hour lockdown for more than
half of the observed data set. Therefore, there is clear evidence that high
temperatures are not able to stop the spread of this virus if there is no
social isolation. Clearly, lockdowns represent the most effective strategy to
prevent the spread of this virus. To the best of our knowledge, this study
describes the first copula model fitted to COVID-19 data. The results of this
study, derived using copula models, are unlike those derived using existing
traditional methods and indicate that the association between COVID-19 and
temperature is weak and no substantial decreases in the number of COVID-19
cases can be expected in response to high temperatures.

## References

  * [1] Huang C, Wang Y, Li X, Ren L, Zhao J, Hu Y, et al.  Clinical features of patients infected with 2019 novel coronavirus in Wuhan, China.  The Lancet. 2020;395(10223):497–506.
  * [2] Kohl A, Chan KH, Peiris JSM, Lam SY, Poon LLM, Yuen KY, et al.  The effects of temperature and relative humidity on the viability of the SARS coronavirus.  Advances in Virology. 2011;2011.  doi:https://doi.org/10.1155/2011/734690.
  * [3] Alghamdi IG, Hussain II, Almalki SS, Alghamdi MS, Alghamdi MM, El-Sheemy MA.  The pattern of Middle East respiratory syndrome coronavirus in Saudi Arabia: a descriptive epidemiological analysis of data from the Saudi Ministry of Health.  International Journal of General Medicine. 2014;7:417.
  * [4] Wang J, Tang K, Feng K, Lin X, Lv W, Chen K, et al.  High Temperature and High Humidity Reduce the Transmission of COVID-19.  Available at SSRN: https://ssrncom/abstract=3551767 or http://dxdoiorg/102139/ssrn3551767. 2020;.
  * [5] Xie J, Zhu Y.  Association between ambient temperature and COVID-19 infection in 122 cities from China.  Science of the Total Environment. 2020;724:138201.  doi:https://doi.org/10.1016/j.scitotenv.2020.138201.
  * [6] Ray EL, Sakrejda K, Lauer SA, Johansson MA, Reich NG.  Infectious disease prediction with kernel conditional density estimation.  Statistics in Medicine. 2017;36(30):4908–4929.
  * [7] Bhatti OI, Do HQ.  Development in copula applications in forestry and environmental sciences.  In: Chandra G, Nautiyal R, Chandra H, editors. Statistical methods and applications in forestry and environmental sciences. Forum for Interdisciplinary Mathematics: Springer, Singapore; 2020. p. 213–230.
  * [8] Alshukairi AN, Zheng J, Zhao J, Nehdi A, Baharoon SA, Layqah L, et al.  High prevalence of MERS-CoV infection in camel workers in Saudi Arabia.  MBio. 2018;9(5).
  * [9] van Doremalen N, Bushmaker T, Munster VJ.  Stability of Middle East respiratory syndrome coronavirus (MERS-CoV) under different environmental conditions.  Eurosurveillance. 2013;18(38).  doi:https://doi.org/10.2807/1560-7917.ES2013.18.38.20590.
  * [10] The General Authority for Statistics (GAStat)  https://www.stats.gov.sa/en/
  * [11] King Abdullah Petroleum Studies and Research Centre  https://datasource.kapsarc.org/pages/eechartbook/
  * [12] Ministry of Health  https://covid19.moh.gov.sa/
  * [13] Weather Underground  https://www.wunderground.com
  * [14] Nelsen RB.  An Introduction to Copulas.  2nd ed. New York: Springer; 2006.
  * [15] Schweizer B, Sklar A.  Probabilistic metric spaces.  Mineola, New York: Courier Corporation; 2011.
  * [16] Sklar A.  Fonctions de répartition á n dimensions et leurs marges.  Publications de lâ€™Institut Statistique de lâ€™Université de Paris. 1959;8:229–231.
  * [17] Cech C.  Copula-based top-down approaches in financial risk aggregation.  University of Applied Sciences Vienna, Working Paper Series No 32. 2006;.
  * [18] Wiboonpongse A, Liu J, Sriboonchitta S, Denoeux T.  Modeling dependence between error components of the stochastic frontier model using copula: application to intercrop coffee production in Northern Thailand.  International Journal of Approximate Reasoning. 2015;65:34–44.
  * [19] Luo J.  Stepwise estimation of D-Vines with arbitrary specified copula pairs and EDA Tools.  Diploma thesis, Technische Universität München, Germany. 2011;.
  * [20] Dißmann J, Brechmann EC, Czado C, Kurowicka D.  Selecting and estimating regular vine copulae and application to financial returns.  Computational Statistics & Data Analysis. 2013;59:52–69.
  * [21] Genest C, Ghoudi K, Rivest LP.  A Semiparametric estimation procedure of dependence parameters in multivariate families of distributions.  Biometrika. 1995;82(3):543–552.
  * [22] Kim G, Silvapulle MJ, Silvapulle P.  Comparison of semiparametric and parametric methods for estimating copulas.  Computational Statistics & Data Analysis. 2007;51(6):2836 – 2850.  doi:https://doi.org/10.1016/j.csda.2006.10.009.
  * [23] Joe H.  Multivariate models and dependence concepts. vol. 73.  London: Chapman & Hall; 1997.
  * [24] Akaike H.  Information theory and an extension of the maximum likelihood principle.  In: Petrov BN, Csaki F, editors. Proceedings of the Second International Symposium on Information Theory. Akademiai Kiado, Budapest; 1973\. p. 267 – 281.
  * [25] Schwarz G, et al.  Estimating the dimension of a model.  The Annals of Statistics. 1978;6(2):461–464.
  * [26] R Development Core Team. R: A language and environment for statistical computing; 2008.  Available from: http://www.R-project.org.
  * [27] Nagler T, Vatter T. rvinecopulib: High performance algorithms for vine copula Modeling; 2020.  Available from: https://CRAN.R-project.org/package=rvinecopulib.


model, including only the right decay pathway. As a result, the left coupler,
filter cavity, and left emitter sideband are not present.

The calculated transmission uses experimentally extracted parameters for the
emitter, coupler, and filter cavity (given in Table 1, Table 2, Table 3). The
drive amplitude $\epsilon$ is calibrated by recording the shift in the coupler
qubit mean frequency while sweeping the RF drive power. For $\Delta/2\pi=$ 805
MHz, drive amplitude is $\epsilon/2\pi=$ 364MHz, yielding $\epsilon/\Delta=$
0.452. At this drive amplitude, $J_{0}({\epsilon}/{\Delta})=$ 0.95,
$J_{1}({\epsilon}/{\Delta})=$ 0.22, and $J_{2}({\epsilon}/{\Delta})=$ 0.03.
The Bessel functions of higher order ($>$ 2) may therefore be safely ignored,
and the input-output matrix equation is truncated after the second-order
negative and positive sidebands ($\bf{H_{0}}$, $\bf{H_{\pm 1}}$, $\bf{H_{\pm
2}}$ are included).