# A First Look into the Carbon Footprint
of Federated Learning

Xinchi Qiu⋆<EMAIL_ADDRESS>
Titouan Parcollet†<EMAIL_ADDRESS>
Javier Fernandez-Marques‡<EMAIL_ADDRESS>
Pedro P. B. Gusmao⋆<EMAIL_ADDRESS>
Daniel J. Beutel§<EMAIL_ADDRESS>
Taner Topal§<EMAIL_ADDRESS>
Akhil Mathur▽<EMAIL_ADDRESS>
Nicholas D. Lane⋆<EMAIL_ADDRESS>⋆Department of Computer Science and
Technology, University of Cambridge
15 JJ Thomson Ave, Cambridge CB3 0FD, United Kingdom †Laboratoire Informatique
d’Avignon, Avignon Université
339 Chemin des Meinajaries, 84000 Avignon, France ‡Department of Computer
Science, University of Oxford
15 Parks Rd, Oxford OX1 3QD, United Kingdom §Adap GmbH
Dorotheenstr. 64, 22301 Hamburg, Germany ▽Nokia Bell Labs
21 JJ Thomson Avenue, Cambridge CB3 0FA, United Kingdom

###### Abstract

Despite impressive results, deep learning-based technologies also raise severe
privacy and environmental concerns induced by the training procedure often
conducted in datacenters. In response, alternatives to centralized training
such as Federated Learning (FL) have emerged. Perhaps unexpectedly, FL, in
particular, is starting to be deployed at a global scale by companies that
must adhere to new legal demands and policies originating from governments and
civil society for privacy protection. However, the potential environmental
impact related to FL remains unclear and unexplored. This paper offers the
first-ever systematic study of the carbon footprint of FL. First, we propose a
rigorous model to quantify the carbon footprint, hence facilitating the
investigation of the relationship between FL design and carbon emissions.
Then, we compare the carbon footprint of FL to traditional centralized
learning. Our findings show that FL, despite being slower to converge in some
cases, may result in a comparatively greener impact than a centralized
equivalent setup. We performed extensive experiments across different types of
datasets, settings and various deep learning models with FL. Finally, we
highlight and connect the reported results to the future challenges and trends
in FL to reduce its environmental impact, including algorithms efficiency,
hardware capabilities, and stronger industry transparency.
Keywords: federated learning, carbon footprint, energy analysis, green AI,
embedded devices

##  1 Introduction

Atmospheric concentrations of carbon dioxide, methane, and nitrous oxide are
at unprecedented levels not seen in the last $800,000$ years (IPCC, 2014).
Together with other anthropogenic drivers, their effects have been detected
throughout client systems and are extremely likely to have been the dominant
cause of the observed global warming since the mid-20th century (Pachauri et
al., 2014; Crowley, 2000). Unfortunately, deep learning (DL) algorithms keep
growing in complexity, and numerous “state-of-the-art” models continue to
emerge, each requiring a substantial amount of computational resources and
energy, resulting in clear environmental costs (Hao, 2019). Indeed, these
models are routinely trained for thousands of hours on specialized hardware
accelerators in datacenters that are extremely energy-consuming (Berriel et
al., 2017). Worse, such a trend is not going to end soon. As Amodei and
Hernandez (2018) show, the amount of compute used by largest ML training has
been exponentially increasing and grown by more than $300,000\times$ from 2012
to 2018, which is equivalent to a 3.4-months doubling period – a rate that
dwarfs the well-known Moore’s 2-year doubling period. This is an observation
that forces us to seriously consider the carbon footprint of deep learning
methods.

The datacenters that enable DL research and commercial operations are not
often accompanied by visual signs of pollution. In a few isolated cases, they
are even powered by environmentally friendly energy sources. Yet, in
aggregate, they are still responsible for an increasingly significant carbon
footprint. Each year they use $200$ terawatt-hours (TWh), which is more than
the national electricity consumption of some countries, representing $0.3\%$
of global carbon emissions (Nature, 2018). In comparison, the entire
information and communications technology ecosystem accounts for only $2\%$.
To put this issue in a more human perspective, each person on average on the
planet is responsible for $5$ tonnes ($11,023$ lbs) of emitted CO2-equivalents
(CO2e) per year (Strubell et al., 2019), while training a large Natural
Language Processing (NLP) transformer model with neural architecture search
may produce $284$ ($626,155$ lbs) tonnes of CO2e (Strubell et al., 2019).

Fortunately, alternatives to datacenter based DL (and other forms of machine
learning) are emerging. Among these, the most prominent to date is Federated
Learning (FL), first formalized by McMahan et al. (2017). Under FL, training
of models primarily occurs in a distributed scenario, either across a large
number of personal devices, such as smartphones (_cross-device_); or across a
small number of institutions that cannot share data among themselves, such as
private hospitals (_cross-silo_). Devices collaboratively learn a global model
but do so without uploading to a datacenter any of the locally stored raw
sensitive data, and a server aggregates all local models following a strategy
such as FedAVG (McMahan and Ramage, 2017; Kairouz et al., 2019; Konečnỳ et
al., 2015). While FL is still a maturing technology, it is already being used
by millions of users on a daily basis; for example, Google uses FL to train
models for: predictive keyboard, device setting recommendation, and hot
keyword personalization on phones (McMahan and Ramage, 2017).

FL was first conceived to address the privacy concerns about sending private
data to centralized data processing. While FL does address these privacy
shortcomings by keeping data local, we currently have little understanding of
its impact on carbon emissions. This is a worrying situation, given the
increasing interest in FL. Therefore, the environmental consequences of FL
need to be assessed before vast systems were deployed and became an everyday
part of life.

This paper addresses this looming gap in the literature. Whereas our still
nascent understanding of the environmental impact of machine learning is
isolated to datacenters – we advance this situation by considering
specifically FL. Existing results (Lacoste et al., 2019; Anthony et al., 2020)
propose approaches that consider CO2e emissions only under centralized
training assumptions. None of these tools generalize to FL which this work
aims to provide: a methodology to properly estimate CO2e emissions related to
FL.

The scientific contributions of this work are as follows:

  * •

Analytical Carbon Footprint Model for FL.  We provide a first-of-its-kind
quantitative CO2e emissions estimation method for FL (Section 3), including
emissions resulting from both hardware training and communication between
server and clients.

  * •

Extensive Experiments. Carbon sensitivity analysis is conducted with this
method on real FL hardwares under different settings, strategies, and tasks
(Section 4). We demonstrate that CO2e emissions depend on a wide range of
hyper-parameters, and emissions from communication between clients and server
can range from $0.4\%$ of total emissions to more than $95\%$. We also show
that by considering more efficient strategies, we can reduce CO2e emissions up
to $60\%$.

  * •

Analysis and Roadmap towards Carbon-friendly FL.  We provide a comprehensive
analysis and discussion of the results to highlight the challenges and future
research directions to develop environmentally-friendly federated learning
(Section 5).

##  2 Federated Learning and the Environment

Federated learning provides many advantages when compared to centralized
learning. At present, data owners are holding more and more sensitive
information, such as individual activity data, life-logging videos, email
conversations, and others (Nishio and Yonetani, 2019). For example, keeping
personal medical and healthcare data private recently became one of the major
ethical concerns (Kish and Topol, 2015). To this extent, and in response to an
increasing number of such privacy issues, policymakers have responded with the
implementation of data privacy legislation such as the European General Data
Protection Regulation (GDPR) (Lim et al., 2020). Due to these regulations,
moving data across national borders becomes subject to data sovereignty law,
making centralized training infeasible in some scenarios (Hsieh et al., 2019).

Furthermore, there are nearly 7 billion connected Internet of Things (IoT)
devices (Lim et al., 2020) and 3 billion smartphones around the world,
potentially giving access to an astonishing amount of training data and
compute power for meaningful research and applications. Using mobile sensing
and smartphones to boost large-scale health studies, such as in Pryss et al.
(2015), Barrett et al. (2020) and Shen (2015), has caused increased interest
in the healthcare research field, and privacy-friendly framework including FL
are potential solutions to answer this demand.

Whilst the carbon footprint for centralized learning has been studied in many
previous works (Anthony et al., 2020; Lacoste et al., 2019; Henderson et al.,
2020; Uchechukwu et al., 2014), the energy consumption and carbon footprint
related to FL remain unexplored. To this extent, this paper proposes to
overcome this issue by giving a first look into the carbon analysis of FL.
State-of-the-art results in deep learning are usually determined by metrics
such as the accuracy of a given model or model size, while energy efficiency
or privacy concerns are often overlooked. Whilst accuracy remains crucial, we
hope to encourage researchers to also focus on other metrics that are in line
with the increasing interest of the civil society for global warming. By
quantifying carbon emissions for FL and demonstrating that a proper design of
the FL setup leads to a decrease of these emissions, we encourage the
integration of the released CO2e as a crucial metric to the FL deployment.

##  3 Quantifying CO2e emissions

Two major steps can be followed to quantify the environmental cost of training
deep learning models either in datacenters or on the edge. First, we perform
an analysis of the energy required by the method (Section 3.1), mostly
accounting for the total amount of energy consumed by the hardware. It
includes training energy for centralized learning and training and
communication energy for FL. Then, the latter amount is converted to CO2e
emissions (Section 3.3) based on geographical locations.

###  3.1 Training Energy Consumption

First, we need to consider the energy consumption coming from GPU and CPU,
which can be measured by sampling GPU and CPU power consumption at training
time (Strubell et al., 2019). For instance, we can repeatedly query the NVIDIA
System Management Interface to sample the GPU power consumption and report the
average over all processed samples while training. Alternatively, we can
estimate using the official hardware power specification or TDP, assuming a
full GPU utilization. Such a use case is not realistic as a GPU is rarely used
at $100$% of its capacity. In the context of FL, not all clients are equipped
with a GPU, and this part can thus be removed from the equation if necessary.
To this extent, we propose to consider $e_{clt}$ as the power of a single
client combining both GPU and CPU measurements. Then, we can connect these
measurements to the total training time of the model.

However, with FL, the wall clock time arises as a challenging estimation to be
done. Indeed, unlike centralized distributed training, FL runs following
communication rounds. During each communication round, certain devices (or
clients) are chosen for training. Since data distribution in clients is
realistically non-IID, the number of communication rounds required is much
higher than for centralized learning. In addition, FL might suffer from system
heterogeneity as different edge devices might not offer the same computational
power (Li et al., 2018). As a matter of fact, such a distribution of clients
is extremely difficult to estimate as sales figures for these devices are not
publicly released by the industry.

We define $T_{FL}(e,N,R)$ to be the total training energy consumption
consisting of a total of $N$ clients in the pool with hardware power $e$ for a
total of $R$ rounds in FL setup:

$\displaystyle
T_{FL}(e,n,R)=\sum_{j=1}^{R}\sum_{i=1}^{N}\mathbb{1}_{\\{Clt_{i,j}\\}}(t_{i}e_{client,i}),$
(1)

where $\mathbb{1}_{\\{Clt_{i,j}\\}}$ is the indicator function indicating if
client $i$ is chosen for training at round $j$, $t_{i}$ the wall clock time
per round and $e_{clt,i}$ the power of client $i$.

It is important to note that other hardware components may also be responsible
for energy consumption, such as RAM or HDD. According to Hodak et al. (2019),
one may expect a variation of around $10\%$ while considering these
parameters. However, they are also highly dependent on the infrastructure
considered and the device distribution that is unfortunately unavailable.

The particular case of cooling in centralized training. Cooling in datacenters
accounts for up to $40\%$ of the total energy consumed (Capozzoli and
Primiceri, 2015). While this parameter does not exist for FL, it is crucial to
consider it when estimating the cost of centralized training. Such estimation
is particularly challenging as it depends on the datacenter efficiency. To
this extent, we propose to use the Power Usage Effectiveness (PUE) ratio. As
reported in the 2019 Data Center Industry Survey Results (UptimeInstitue,
2019), the world average PUE for the year 2019 is $1.67$. As expected,
observed PUE strongly varies depending on the considered company. For
instance, Google declares a comprehensive trailing twelve-month PUE ratio of
$1.11$ (Google, 2020) compared to $1.2$ and $1.125$ for Amazon (AWS, 2020) and
Microsoft (Microsoft, 2015) respectively. Therefore, Eq. (1) is extended to
centralized training as:

$T_{center}=PUE(te_{center}),$ (2)

with $e_{center}$ the power combining both GPUs and CPUs in centralized
training setup, and $t$ stands for the total training time. In addition, the
cost of transferring the model parameters from the RAM to the VRAM is
negligible.

###  3.2 Wide-area-networking (WAN) Emission

As clients continue to perform individual training on local datasets, their
models begin to diverge. To mitigate this effect, model aggregation must be
performed by the server in a process that requires frequent exchange in model
parameters.

For this reason, we must also consider the amount of emission derived from
data exchange between server and clients during the beginning and the end of a
round.

According to Malmodin and Lundén (2018), the embodied carbon footprint for
Information and Communication Technology (ICT) network operators is mainly
related to the construction and deployment of the network infrastructure
including digging down cable ducts and raising antenna towers.

Regarding FL, we estimate the energy required to transferring model parameters
between the server and the clients following two parts. The first part is the
download and upload energy consumed by routers, while the second part is the
energy consumed by the hardware when downloading and uploading the model
parameters. We propose to use country-specific download and upload speed as
reported on Speedtest (Speedtest, ) and router power reported on The Power
Consumption Database (Database, ). Due to the rapid development of ICT, we
propose to use the median power obtained from all data submitted during 2020
from the database. We also take idle power consumption of hardware into
consideration when they communicate model weights.

Let us define $D$ and $U$ the download and upload speeds expressed in Mbps
respectively. Then let $S$ be the size of the model in Mb, $e_{r}$ be the
power of the router, and $e_{idle}$ be the power of the hardware of the
clients when it is in idle. Therefore, the communication energy per round is
defined as:

$C(e,N,R)=\sum_{j=1}^{R}\sum_{i=1}^{N}\mathbb{1}_{\\{Clt_{i,j}\\}}S\left(\frac{1}{D}+\frac{1}{U}\right)(e_{r}+e_{idle,i}).$
(3)

###  3.3 Converting to CO2e emissions

Realistically, it is difficult to compute the exact amount of CO2e emitted in
a given location since the information regarding the energy grid, i.e., the
conversion rate from energy to CO2e, is rarely publicly available (Lacoste et
al., 2019; Hodak et al., 2019). Therefore, we assume that all datacenters and
the edge devices are connected to their local grid directly linked to their
physical location. Electricity-specific CO2e emission factors are obtained
from Brander et al. (2011). Out of all these conversion factors expressed in
kg CO2e/kWh, we picked three of the most representative ones: France
($0.0790$), USA ($0.5741$) and China ($0.9746$). The estimation methodology
provided takes into accounts both transmission and distribution emission
factors (i.e. energy lost when transmitting and distributing electricity) and
the efficiency of heat plants. As expected, countries relying on carbon-
efficient productions are able to lower their corresponding emission factor
(e.g. France, Canada).

Therefore, the total amount of CO2e emitted in kilograms for federated
learning ($E_{FL}$) and centralized training ($E_{center}$) obtained from Eq.
1, 2 and 3 are:

$\displaystyle E_{FL}=c_{rate}[T(e,N,R)+C(e,N,R)],$ (4) $\displaystyle
E_{center}=c_{rate}T_{center},$ (5)

where $c_{rate}$ is the conversion factor. Carbon emissions may be compensated
by carbon offsetting or with the purchases of Renewable Energy Credits (RECs,
in the USA) or Tradable Green Certificates (TGCs, in the EU). Carbon
offsetting allows polluting actions to be compensated directly via different
investments in environment-friendly projects, such as renewable energies or
massive tree planting Anderson (2012). RECs and TGCs Bertoldi and Huld (2006),
on the other hand, guarantee that a specific volume of electricity of their
owner is generated from renewable energy sources. More precisely, RECs and
TGCs purchasing aims to create more renewable energy in the long-term
according to the additionality principle, by creating an increased demand for
these energy sources. In the specific context of CO2 estimates, RECs and TGCs
enable institutions and companies to voluntarily decrease their individual
$c_{rate}$. However, in our analysis, carbon rates are obtained at country
level and do not integrate industry level carbon offsetting schemes or RECs.

##  4 Experiments

This paper provides extensive estimates across different type of tasks and
datasets, including image classification with CIFAR10 Krizhevsky et al.
(2009), and ImageNet Russakovsky et al. (2015), and keyword spotting with
Speech Commands Warden (2018). First, we provide an estimate of the carbon
footprint following different realistic FL setups. Then, we conduct an in-
depth analysis of these results to highlight the potential gains observed with
FL.

###  4.1 Experimental Protocol

In addition to the carbon model (Section 3), results are influenced by
configurations of the hardware and systems of datacenters and FL respectively.
The FL pipeline is implemented using the Flower framework (Beutel et al.,
2020) and PyTorch (Paszke et al., 2019).

Centralized training hardware. We run our experiments on a server equipped
with a 2x Xeon 6152 22-core processor and NVIDIA Tesla V100 32GB GPUs. The CPU
and GPU have a TDP of 240W and 250W respectively. We use a single GPU per
experiment and, measure the power drawn by both CPU and GPU by making use of
NVIDIA-smi monitoring and the cross-platform psutil tools.

Federated learning hardware. We propose to use of NVIDIA Tegra X2 (Smith,
2017) and Jetson Xavier NX Smith (2019) devices as our FL clients. These
devices can be viewed as a realistic pool of FL clients since they can be
found embedded in various IoT devices including cars, smartphones, video game
consoles. NVIDIA Tegra X2 offers two power modes with theoretical power limits
of $7.5W$ and $15W$ and Xavier NX offers $10W$ and $15W$. Across our different
runs, we use the lower power mode for each device, and we employ the built-in
utility tegrastats to report the overall power consumption. For both power
consumption and training time, we report averaged values across several FL
rounds for each experiment. We also measure the idle power consumption for
both devices, which was recorded as $1.35W$ and $2.25W$ for TX2 and NX
respectively.

Datasets. We conduct our estimations on two image classification tasks of
different complexity both in terms of the number of samples and classes:
CIFAR10 (Krizhevsky et al., 2009) and ImageNet (Russakovsky et al., 2015). In
addition, we also perform analysis on the Speech Commands dataset (Warden,
2018) which consists of $65K$ 1-second long audio clips of $30$ keywords, with
each clip consisting of only one keyword. Following the setup described in
Zhang et al. (2017), we train the model to classify the audio clips into one
of the $10$ keywords - “Yes”, “No”, “Up”,“Down”, “Left”, “Right”, “On”, “Off”,
“Stop”, “Go”, along with “silence” (i.e. no word spoken) and “unknown” word,
representing the remaining $20$ keywords from the dataset. The training set
contains a total of $56,196$ clips with $32,550$ ($57\%$) samples from the
“unknown” class and around $1800$ samples ($3.3\%$) from each of the remaining
classes, hence the dataset is naturally unbalanced.

Model Architectures. For image classification tasks, FL and centralized
training rely on an identically implemented ResNet-18 architecture to
alleviate any variations. These models are trained with SGD but only the
centralized setting makes use of momentum. For the sake of completeness, we
choose to use different deep learning model for the Speech Commands dataset.
We employ $4$ layers of LSTM each with $256$ nodes. The models are trained
using Adam optimization. Also, the hyper-parameters, such as learning rates,
are set to be the same as centralized learning without further tuning.

Data partition methodology. Typical, FL settings can be classified as either
_cross-silo_ or _cross-device_. In a _cross-silo_ scenario, clients are
generally few, with high availability during all rounds, and are likely to
have similar data distribution for training, e.g. consortium of hospitals.
This scenario serves as motivation to consider Independent and Identically
Distributed (IID) distributions.

On the other hand, a _cross-device_ system will likely encompass thousands of
clients having very different data distributions (non-IID) participating in
just a few rounds, e.g. training of next-word prediction models on mobile
devices.

We simulate the difference in data distribution in these two scenarios using
two different partition schemes: a uniform partition (IID) where each client
has approximately the same proportion of each class from the original dataset,
and a heterogeneous partition (non-IID) for which each client has an
unbalanced and different proportion of each class. We follow the latent
Dirichlet allocation (LDA) partition method Reddi et al. (2020); Yurochkin et
al. (2019); Hsu et al. (2019) ensuring each client get allocated the same
number of training samples. Each sample is drawn independently with class
labels following a categorical distribution over $N$ classes parameterized
with a vector q ($q_{i}\geq 0$, $i\in[1,m]$ and $\sum q_{i}=1$ for a total of
$m$ classes from the dataset). Thus, to simulate the partition, we draw
$\textbf{q}\sim Dir(\alpha\textbf{p})$ from a Dirichlet distribution, where p
stands for the prior distribution of the dataset, and $\alpha$ stands for the
concentration which controls the level of heterogeneity of the partition. As
$\alpha\to\infty$, the partition becomes more uniform (IID), and as $\alpha\to
0$, the partition becomes more heterogeneous. As the dataset is balanced
across classes for both CIFAR10 and ImageNet, the prior distribution p is
uniform. For ImageNet, we chose $\alpha=1000$ for the IID dataset partition
and $\alpha=0.5$ for non-IID following Yurochkin et al. (2019) and Hsu et al.
(2019). For CIFAR10, we choose $\alpha=0.1$ following the same protocol as
Reddi et al. (2020). As for Speech Commands, in light of the unbalanced nature
of the dataset, we propose to change the prior of LDA from uniform
distribution to multinomial distribution. Hence the LDA can be summarized as:

p
$\displaystyle=\left(\frac{N_{1}}{N},\frac{N_{2}}{N},...,\frac{N_{m}}{N}\right)$
(6) q $\displaystyle\sim Dir(\alpha\textbf{p}),$ (7)

where $N_{i}$ stands for the number of data from class $i$, $N$ stands for
total number of data in the dataset. According to Yurochkin et al. (2019); Hsu
et al. (2019) $\alpha$ is commonly set as $0.5$ for a non-IID partition of
balanced dataset. Given the aforementioned unbalanced nature of the dataset,
we propose to match the variance of $10$ keywords classes with multinomial
prior to the variance of $10$ keywords classes with a uniform prior by
changing $\alpha$ to $1.0$.

Client pool. Following Reddi et al. (2020) we propose to compose a pool of
$500$ client for CIFAR10 with $10$ active clients training concurrently. Then,
to investigate the impact of a different number of available clients, we
conducted both ImageNet and SpeechCommands tasks with a set of $100$ clients
with $10$ sampled clients each round.

FL strategy. To better reflect realistic FL scenarios, we propose to
investigate the energy consumption with the common FedAVG strategy (McMahan et
al., 2017), and more complex FedADAM strategy (Reddi et al., 2020). For
CIFAR10, we follow the experimental protocol proposed in Reddi et al. (2020)
considering the suggested best values for $\eta$, $\eta_{l}$, and $\tau$ in
almost every experiment except for FedAVG, where we had to lower the value of
$\eta_{l}$ to $10^{-3/2}$ to allow training. All other experiments used a
server learning rate $\eta=0.1$ and $\tau=0.001$.

Local epoch (LE). We also propose to vary the number of local epochs done on
each client to better highlight the contribution of the local computations to
the total emissions. To be consistent, we choose to do $1$ and $5$ local
epochs across all tasks.

Target accuracies. As our primary objective is to estimate the carbon
footprint, we leave reaching state-of-the-art performance as a secondary
objective. This is a common procedure when evaluating FL workloads. For both
CIFAR10 and ImageNet, we set this threshold to $50\%$ top-1 accuracy. For
Speech Commands, the threshold is set to $70\%$.

###  4.2 Experimental Results

This section presents the number of rounds required by each setup to reach its
target accuracy as well as the energy associated with both centralized and FL.

Power consumption and training times obtained for all FL and centralized
setups are reported in Table 1, while the numbers of communication rounds
required by each setup to reach their target accuracies are summarized in
Table 2.

|  |  |  |  |  | Costs to Reach Threshold Accuracy
---|---|---|---|---|---|---
Dataset |  Training
Strategy |  HW |  Power
Usage
(W) |  Local
Epochs |  Time per Epoch(s) |  Num.
Rounds |  Time(s) | Energy | Total
per device | Energy
|  |  |  |  |  |  |  | (Wh) | (Wh)
CIFAR10 | Centralized | V100 | 160+42 | 1 | 24 | 2 | 48 | 2.7 | 2.7
FedAVG | TX2 | 4.7 | 1 | 0.8 | 1248 | 998 | 1.3 | 13
NX | 6.3 | 0.6 | 748 | 1.3 | 13
FedAdam | TX2 | 4.7 | 1 | 0.8 | 674 | 539 | 0.7 | 7
NX | 6.3 | 0.6 | 404 | 0.7 | 7
ImageNet | Centralized | V100 | 220+84 | 1 | 1,440 | 8 | 11,500 | 971 | 971
FedAVG | TX2 | 6.5 | 1 | 474 | 232 | 109,968 | 198 | 1,985
NX | 9.7 | 273 | 63,336 | 170 | 1,706
Speech Commands | Centralized | V100 | 68+56 | 1 | 52 | 6 | 312 | 10.7 | 10.7
FedAVG | TX2 | 5.7 | 5 | 1.6 | 119 | 952 | 1.5 | 15
NX | 7.9 | 0.9 | 536 | 1.5 | 15
FedAdam | TX2 | 5.7 | 1 | 1.6 | 140 | 224 | 0.4 | 4
NX | 7.9 | 0.9 | 126 | 0.3 | 3

Table 1: Comparing energy consumption of centralized training using data-center GPUs against FL settings where each client trains on a small dataset partition using low-power GPU-enabled edge devices. For FL rows, each strategy reports the setting (1 or 5 local epochs) that lead to reaching the target accuracy faster. For centralized setting, 1 Local Epoch represents a standard epoch using the entire dataset and, power usage is reported as GPU+CPU measured wattage. For FL rows, the total energy is obtained by multiplying by the number of clients participating in each round (set to 10 in all our experiments). Despite edge devices consuming an order of magnitude less power, the total energy required for FL is often greater (but of the same order of magnitude) than centralized training. For Speech Commands, a very lightweight workload, FL can reach the target accuracy while requiring little energy. Dataset | Training | Local | Partition
---|---|---|---
Strategy | Epochs | IID | non-IID
CIFAR10 | FedAVG | 1 | 1248 |  $>$5000
5 | 427 | 4955
FedAdam | 1 | 674 | 4788
5 | 349 | 1952
ImageNet | FedAVG | 1 | 232 | 339
5 | 95 | 114
Speech Commands | FedAVG | 1 |  $>$1000 | 770
5 | 119 | 140
FedAdam | 1 | 140 | 193
5 | 53 | 66
Table 2: Number of FL rounds needed for each dataset-strategy pair to reach
the threshold accuracy when data is partitioned in IID and non-IID fashion. We
observe that increasing the number of local epochs always results in fewer FL
rounds to reach convergence. However, this does not guarantee a smaller
overall energy consumption. The adaptive aggregation strategy FedADAM
consistently results in faster global model convergence.

It is worth noting that centralized training (V100) took solely $2$ epochs to
achieve the target accuracy for CIFAR10, and $8$ epochs for ImageNet. This
translates to $48$ seconds for CIFAR10 and $3.2$ hours for ImageNet.
Conversely, standard FedAVG failed to converge within the allotted $5000$
rounds in the non-IID case using only $1$ local epoch for CIFAR10, while a
more efficient strategy (i.e FedADAM) was able to reach the target.

CIFAR10 | Centr. | IID 5LE | non-IID 1LE | non-IID 5LE
---|---|---|---|---
Country/ | PUE | PUE | FedAVG | FedADAM | FedAVG | FedADAM | FedAVG | FedADAM
CO2e(g) | $1.67$ | $1.11$ | TX2 | NX | TX2 | NX | TX2 | NX | TX2 | NX | TX2 | NX | TX2 | NX
USA | 2.5 | 1.6 | 48.4 | 52.7 | 39.6 | 43.0 |  $>$452 |  $>$502 | 433 | 481 | 562 | 611 | 221 | 241
China | 4.4 | 2.9 | 110 | 121 | 90.3 | 98.8 |  $>$1090 |  $>$1210 | 1043 | 1160 | 1281 | 1400 | 505 | 553
France | 0.4 | 0.2 | 5.1 | 5.4 | 4.1 | 4.5 |  $>$42.7 |  $>$47.2 | 40.9 | 45.2 | 58.7 | 63.2 | 23.1 | 24.9

ImageNet | Centralized | IID 5LE | non-IID 1LE | non-IID 5LE
---|---|---|---|---
Country/ | $PUE=1.67$ | $PUE=1.11$ | FedAVG | FedAVG | FedAVG
CO2e(g) | V100 | V100 | TX2 | NX | TX2 | NX | TX2 | NX
USA | 889 | 591 | 2232 | 1921 | 1616 | 1396 | 2679 | 2305
China | 1583 | 1052 | 3982 | 3427 | 2898 | 2509 | 4778 | 4113
France | 129 | 85 | 322 | 277 | 232 | 200 | 386 | 332

SpeechCmd | Centralized | IID 5LE | non-IID 1LE | non-IID 5LE
---|---|---|---|---
Country/ | $PUE=1.67$ | $PUE=1.11$ | FedAVG | FedADAM | FedAVG | FedADAM | FedAVG | FedADAM
CO2e(g) | V100 | V100 | TX2 | NX | TX2 | NX | TX2 | NX | TX2 | NX | TX2 | NX | TX2 | NX
USA | 9.8 | 6.5 | 13.1 | 11.9 | 5.7 | 5.2 | 42.3 | 43.6 | 10.6 | 10.9 | 15.5 | 14.0 | 7.3 | 6.6
China | 17.5 | 11.6 | 26.7 | 24.8 | 11.7 | 10.8 | 96.5 | 101 | 24.2 | 25.4 | 31.4 | 29.2 | 14.8 | 13.8
France | 1.4 | 0.9 | 1.6 | 1.4 | 0.7 | 0.6 | 4.4 | 4.4 | 1.1 | 1.1 | 1.9 | 1.7 | 0.9 | 0.8

Table 3: CO2e emissions (expressed in grams, i.e lower is better) for both
centralized learning and FL when they reach the target accuracies, with
different setups. The tables report results of both FedAvg and FedADAM in both
IID and non-IID partitions. As non-IID is more realistic, we report both $1$
and $5$ local epochs experiment results for this setup only.

For Speech Commands experiments, we can definitely see more fluctuations with
the test accuracy in the non-IID case as the dataset is naturally unbalanced,
hence more rounds are required to reach the target test accuracy. It is
interesting that FedAVG needs even more rounds for IID than non-IID if we only
do one local epoch, which might again due to the fact that the dataset is
naturally unbalanced and we keep all the hyper-parameters the same even with
different setup, which shows the importance of hyper-parameters tuning.

Also, by applying FedADAM, the number of communication rounds is consistently
lower across all datasets compared with FedAVG, especially for non-IID
partitions with a minimal decrease of $50\%$.

##  5 Carbon Footprint of Federated Learning

So far we have considered the energy required to achieve a given accuracy on
different tasks for various sets of hyper-parameters and optimizers. We now
turn our attention to how this translates into carbon emissions.

The first thing to notice is that, due to the large difference between
electricity-specific CO2e emission factors among countries, carbon footprint
of both centralized training and FL can be highly dependent on the geolocation
of hardwares. Training in France always has lowest CO2e emissions if we
compare the same training setup due to the fact that France has nuclear energy
has their main source of and hence the lowest energy to CO2e conversion rate.

Secondly, CO2e emissions induced by FL highly depend on other factors as well,
such as DL task and model architecture. Indeed, we show that a basic FL setup
relying on FedAVG clearly emits more carbon for image-based tasks using
ResNet-18 compared to modern GPUs and centralized training, while it also
emits less carbon in some setups with the Speech Commands dataset using a
stacked LSTM model. As a matter of fact, the lower CO2e estimates obtained
with FL on Speech Commands are of great interest. Indeed, in practice, it is
most likely that deployed FL tasks (i.e on device) will be lightweight and
certainly aiming at lowering the number of communications. Hence, this type of
realistic FL use-cases could benefit to our planet.

Hardware efficiency is also a critical factor when estimating the total carbon
footprint. As new AI applications for consumers are created everyday, it is
realistic to assume that chips like Tegra X2 will soon be embedded in numerous
devices including smartphones, tablets, and others. However, such specialized
hardware is certainly not an exact estimate of what is currently being used
for FL. Therefore, to facilitate environmental impact estimations of large
scale FL deployment, the industry must increase its transparency with respect
to their devices distribution over the market.

Also, as explained in our estimation methodology, FL will always have an
advantage due to the cooling needs of datacenters. In fact, and even though
GPUs or even TPUs are getting more efficient in terms of computational power
delivered by the amount of energy consumed, the need for a strong and energy-
consuming cooling remains, thus the FL can always benefit more from the
hardware advancement.

$0$$10$$20$$30$$40$$50$$10^{0}$$10^{1}$$10^{2}$Number of centralized epochlog(CO2e emitted in g)(a) CIFAR10FL (5 local epochs)FL (1 local epochs)NVIDIA Tesla V100Breakeven line |  $0$$10$$20$$30$$40$$50$$10^{2}$$10^{3}$Number of centralized epochlog(CO2e emitted in g)(b) ImageNet |  $0$$20$$40$$60$$80$$10^{0}$$10^{1}$$10^{2}$Number of centralized epochlog(CO2e emitted in g)(c) Speech Commands
---|---|---

Figure 1: Growth of CO2e emissions (in log scale) with a $PUE=1.67$ for
centralized learning and TX2 devices for FL in the USA (expressed in grams,
i.e lower is better) . Communication rounds in FL are converted to centralized
epochs for a fair comparison, and CO2e emissions are linearly dependent on the
number of centralized epoch. (a) For CIFAR10, 1 centralized epoch is
equivalent to 50 communication rounds for 1 Local Epoch (LE) and 10 for 5 LE,
as there are a total of $500$ clients. The green line shows that with an equal
amount of emissions between FL and V100, FL would train for $21$ rounds with
5LE and $27$ rounds with 1LE. (b) For ImageNet and Speech Commands and due to
the smaller size of the total client pool ($100$), 1 centralized epoch is
equivalent to 2 communication rounds with 5LE and 10 rounds with 1LE. The
green line shows that with an equal amount of emissions, FL can only train for
$190$ rounds with 1LE and $38$ rounds with 5LE. (c) For Speech Commands, if
centralized learning emits the same amount of CO2e as FL 5LE with FedADAM in
non-IID partition, centralized learning can only train for $4$ epoch reaching
$60\%$ of testing accuracy.

Furthermore, CO2 emission depends on the distribution of clients’ datasets.
Our results show that realistic training conditions for FL (i.e. non-IID data)
are largely responsible for longer training times, which in turn translates to
high level of CO2e emissions. While it is well known that the simpler
aggregation form of FL (e.g. FedAVG) performs reasonably well on IID data, it
definitely struggles with non-IID partitioned data in terms of accuracy (Li et
al., 2018; Qian et al., 2020). Interestingly, more complex strategies such as
FedADAM can enable a decrease of up to $60\%$ and $75\%$ of the emitted CO2e
on CIFAR10 and Speech Commands respectively compared to FedAVG. It is worth
pointing out that for both CIFAR10 and Speech Commands non-IID partitions, 1LE
produces more carbon than 5LE regardless of aggregation strategy or devices.
It is because the communication consumption plays a big role in the total
energy consumption, and with 5LE communication cost less than with 1LE because
less number of communication rounds required.

As it is the case in centralized training, hyper-parameter tuning is of great
importance in reducing training times. In our experiments, we set the hyper-
parameters the same as centralized training in each task, but further tuning
can be done to make FL converge faster. Also for FL, tuning becomes a more
arduous task since it involves the tuning of potentially hundreds of different
models (i.e. local models in the clients), each making use of small datasets
which likely follow a very skewed distribution. In addition to client-side
tuning, the aggregation strategy (e.g. FedADAM) might also offer further
parameterization, therefore increasing the complexity of the tuning process.
Therefore, novel algorithms should carefully be designed to minimize the
carbon emission by jointly maximizing the accuracy and minimizing the released
CO2e.

Figure 5 shows the growth of carbon emission when the number of centralized
epoch increases. We first see that for CIFAR10, FL with 1LE has the highest
slope, while for the other two datasets, centralized learning have the highest
slope. Normally centralized learning should have highest slope as TDPs of
centralized learning hardware are much higher than for FL. However, for
CIFAR10, as the model size is big, communication consumption is much bigger
than the actual training consumption, resulting very high slope, which tells
us that applying a complicated model does not always benefit.

While it is clear that FedADAM improves the convergence rate, hence reducing
the overall emission, it is still clearly not as “green” as centralized
learning for image tasks. For CIFAR10, if FL was constrained to an equivalent
level of carbon emissions as V100, it would only be able to train for $21$
rounds with $5$ local epochs and $27$ rounds with $1$, resulting in degraded
performances even with FedADAM. The same goes for ImageNet. Indeed, FL would
only train for $190$ rounds with 1 LE and $38$ rounds with 5. On the other
hand, with Speech Commands, FL outperforms centralized learning in certain
setups. With the same CO2e as FL with 5LE using FedADAM in non-IID partition,
V100 would only train for less than $4$ epochs (i.e. $60\%$ of test accuracy).
It is interesting to note that, for ImageNet, the communication cost is
negligible, hence both FL curves become very similar.

Finally, it is also worth noticing that the percentage of CO2e emission
resulting from WAN changes between dataset and FL setups. It highly depends on
the size of model, size of the dataset in each client and, the energy consumed
by clients during training. More precisely, communications accounted for
average for up to $0.4\%$ (ImageNet with 5 local epochs) and $95\%$ (CIFAR10
with 1 local epoch) in the total emissions. With CIFAR10 tasks, communication
actually emits much more CO2e than training, while on the other hand, WAN
plays a very small role for ImageNet.

##  6 Conclusion

Climate change is real, and deep learning plays an increasing role in this
tragedy. Fortunately, a number of recent studies have begun to detail the
environmental costs of their novel deep learning methods, sometimes even
integrating CO2e emissions as an objective to be minimized. Following this
important trend, this paper takes a first look into the carbon footprint of an
increasingly deployed training strategy known as federated learning. In
particular, this work introduces a generalized methodology to systematically
compute the carbon footprint of any federated learning setups, and conducts an
extensive experiments on real FL hardwares under different settings, models,
strategies and tasks. We highlight the carbon footprint of FL with different
perspectives and demonstrates that each element of FL can have an impact on
the total CO2e emission, including physical location, deep learning tasks,
model architecture, FL aggregation strategy and hardware efficiency. Finally,
we hope to emphasize the importance of taking environment impact into
consideration for future research, and innovative research for both deep
learning and federated learning can integrate the carbon footprint as a novel
metric.

Acknowledgments

This work was supported by the UK’s Engineering and PhysicalSciences Research
Council (EPSRC) with grants EP/M50659X/1 and EP/S001530/1 and the European
Research Council via the REDIAL project.

## References

  * Amodei and Hernandez (2018) D. Amodei and D. Hernandez.  Ai and compute.  _Heruntergeladen von https://blog. openai. com/aiand-compute_ , 2018\.
  * Anderson (2012) K. Anderson.  The inconvenient truth of carbon offsets.  _Nature_ , 484(7392):7–7, 2012.
  * Anthony et al. (2020) L. F. W. Anthony, B. Kanding, and R. Selvan.  Carbontracker: Tracking and predicting the carbon footprint of training deep learning models.  _arXiv preprint arXiv:2007.03051_ , 2020.
  * AWS (2020) AWS.  Aws and sustainability.  https://aws.amazon.com/about-aws/sustainability/, 2020.
  * Barrett et al. (2020) E. K. Barrett, C. M. Fard, H. N. Katinas, C. V. Moens, L. E. Perry, B. E. Ruddy, S. D. Shah, I. S. Tucker, T. J. Wilson, M. Rucker, L. Cai, L. E. Barnes, and M. Boukhechba.  Mobile sensing: Leveraging machine learning for efficient human behavior modeling.  In _2020 Systems and Information Engineering Design Symposium (SIEDS)_ , pages 1–7, 2020.  doi: 10.1109/SIEDS49339.2020.9106648.
  * Berriel et al. (2017) R. F. Berriel, A. T. Lopes, A. Rodrigues, F. M. Varejão, and T. Oliveira-Santos.  Monthly energy consumption forecast: A deep learning approach.  In _2017 International Joint Conference on Neural Networks (IJCNN)_ , pages 4283–4290, 2017.
  * Bertoldi and Huld (2006) P. Bertoldi and T. Huld.  Tradable certificates for renewable electricity and energy savings.  _Energy policy_ , 34(2):212–222, 2006.
  * Beutel et al. (2020) D. J. Beutel, T. Topal, A. Mathur, X. Qiu, T. Parcollet, and N. D. Lane.  Flower: A friendly federated learning research framework, 2020.
  * Brander et al. (2011) M. Brander, A. Sood, C. Wylie, A. Haughton, and J. Lovell.  Technical paper— electricity-specific emission factors for grid electricity.  _Ecometrica, Emissionfactors. com_ , 2011.
  * Capozzoli and Primiceri (2015) A. Capozzoli and G. Primiceri.  Cooling systems in data centers: state of art and emerging technologies.  _Energy Procedia_ , 83:484–493, 2015.
  * Crowley (2000) T. J. Crowley.  Causes of climate change over the past 1000 years.  _Science_ , 289(5477):270–277, 2000.
  * (12) T. P. C. Database.  http://www.tpcdb.com/list.php?page=1&type=11.
  * Google (2020) Google.  Efficiency-data centres.  https://www.google.co.uk/about/datacenters/ efficiency/, 2020\.
  * Hao (2019) K. Hao.  Training a single ai model can emit as much carbon as five cars in their lifetimes.  _MIT Technology Review_ , 2019.
  * Henderson et al. (2020) P. Henderson, J. Hu, J. Romoff, E. Brunskill, D. Jurafsky, and J. Pineau.  Towards the systematic reporting of the energy and carbon footprints of machine learning.  _arXiv preprint arXiv:2002.05651_ , 2020.
  * Hodak et al. (2019) M. Hodak, M. Gorkovenko, and A. Dholakia.  Towards power efficiency in deep learning on data center hardware.  In _2019 IEEE International Conference on Big Data (Big Data)_ , pages 1814–1820. IEEE, 2019.
  * Hsieh et al. (2019) K. Hsieh, A. Phanishayee, O. Mutlu, and P. B. Gibbons.  The non-iid data quagmire of decentralized machine learning.  _arXiv preprint arXiv:1910.00189_ , 2019.
  * Hsu et al. (2019) T.-M. H. Hsu, H. Qi, and M. Brown.  Measuring the effects of non-identical data distribution for federated visual classification.  _arXiv preprint arXiv:1909.06335_ , 2019.
  * IPCC (2014) IPCC.  Climate change 2014 synthesis report.  2014\.
  * Kairouz et al. (2019) P. Kairouz, H. B. McMahan, B. Avent, A. Bellet, M. Bennis, A. N. Bhagoji, K. Bonawitz, Z. Charles, G. Cormode, R. Cummings, et al.  Advances and open problems in federated learning.  _arXiv preprint arXiv:1912.04977_ , 2019.
  * Kish and Topol (2015) L. J. Kish and E. J. Topol.  Unpatients—why patients should own their medical data.  _Nature biotechnology_ , 33(9):921–924, 2015\.
  * Konečnỳ et al. (2015) J. Konečnỳ, B. McMahan, and D. Ramage.  Federated optimization: Distributed optimization beyond the datacenter.  _arXiv preprint arXiv:1511.03575_ , 2015.
  * Krizhevsky et al. (2009) A. Krizhevsky, G. Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * Lacoste et al. (2019) A. Lacoste, A. Luccioni, V. Schmidt, and T. Dandres.  Quantifying the carbon emissions of machine learning.  _arXiv preprint arXiv:1910.09700_ , 2019.
  * Li et al. (2018) T. Li, A. K. Sahu, M. Zaheer, M. Sanjabi, A. Talwalkar, and V. Smith.  Federated optimization in heterogeneous networks.  _arXiv preprint arXiv:1812.06127_ , 2018.
  * Lim et al. (2020) W. Y. B. Lim, N. C. Luong, D. T. Hoang, Y. Jiao, Y.-C. Liang, Q. Yang, D. Niyato, and C. Miao.  Federated learning in mobile edge networks: A comprehensive survey.  _IEEE Communications Surveys & Tutorials_, 2020.
  * Malmodin and Lundén (2018) J. Malmodin and D. Lundén.  The energy and carbon footprint of the global ict and e&m sectors 2010–2015.  _Sustainability_ , 10(9):3027, 2018.
  * McMahan and Ramage (2017) B. McMahan and D. Ramage.  Federated learning: Collaborative machine learning without centralized training data.  _Google Research Blog_ , 3, 2017.
  * McMahan et al. (2017) B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas.  Communication-Efficient Learning of Deep Networks from Decentralized Data.  In A. Singh and J. Zhu, editors, _Proceedings of the 20th International Conference on Artificial Intelligence and Statistics_ , volume 54 of _Proceedings of Machine Learning Research_ , pages 1273–1282, Fort Lauderdale, FL, USA, 20–22 Apr 2017. PMLR.  URL http://proceedings.mlr.press/v54/mcmahan17a.html.
  * Microsoft (2015) Microsoft.  Datacenter fact sheet - microsoft download center.  http://download.microsoft.com/download/8/2/9/8297f7c7-ae81-4e99-b1db-d65 a01f7a8ef/microsoft_cloud_infrastructure_datacenter_and_network_fact_ sheet.pdf, 2015\.
  * Nature (2018) Nature.  How to stop data centres from gobbling up the world’s electricity.  https://www.nature.com/articles/d41586-018-06610-y, Sep 2018.
  * Nishio and Yonetani (2019) T. Nishio and R. Yonetani.  Client selection for federated learning with heterogeneous resources in mobile edge.  In _ICC 2019 - 2019 IEEE International Conference on Communications (ICC)_ , pages 1–7, 2019.
  * Pachauri et al. (2014) R. K. Pachauri, L. Gomez-Echeverri, and K. Riahi.  Synthesis report: summary for policy makers.  2014\.
  * Paszke et al. (2019) A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems 32_ , pages 8024–8035. Curran Associates, Inc., 2019.
  * Pryss et al. (2015) R. Pryss, M. Reichert, J. Herrmann, B. Langguth, and W. Schlee.  Mobile crowd sensing in clinical and psychological trials – a case study.  In _2015 IEEE 28th International Symposium on Computer-Based Medical Systems_ , pages 23–24, 2015.
  * Qian et al. (2020) J. Qian, X. Fafoutis, and L. K. Hansen.  Towards federated learning: Robustness analytics to data heterogeneity.  _arXiv preprint arXiv:2002.05038_ , 2020.
  * Reddi et al. (2020) S. Reddi, Z. Charles, M. Zaheer, Z. Garrett, K. Rush, J. Konečnỳ, S. Kumar, and H. B. McMahan.  Adaptive federated optimization.  _arXiv preprint arXiv:2003.00295_ , 2020.
  * Russakovsky et al. (2015) O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, et al.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ , 115(3):211–252, 2015.
  * Shen (2015) H. Shen.  Smartphones set to boost large-scale health studies.  _Nature News_ , 2015.
  * Smith (2017) R. Smith.  Nvidia announces jetson tx2: Parket comes to nvidia’s embedded system kit.  https://www.anandtech.com/show/11185/ia-announces-jetson-tx2-parker, mar 2017.
  * Smith (2019) R. Smith.  Nvidia gives jetson agx xavier a trim, announces nano-sized jetson xavier nx.  https://www.anandtech.com/show/15070/nvidia-gives-jetson-xavier-a-trim -announces-nanosized-jetson-xavier-nx, nov 2019.
  * (42) Speedtest.  https://www.speedtest.net/global-index.
  * Strubell et al. (2019) E. Strubell, A. Ganesh, and A. McCallum.  Energy and policy considerations for deep learning in nlp.  _arXiv preprint arXiv:1906.02243_ , 2019.
  * Uchechukwu et al. (2014) A. Uchechukwu, K. Li, Y. Shen, et al.  Energy consumption in cloud computing data centers.  _International Journal of Cloud Computing and Services Science (IJ-CLOSER)_ , 3(3):145–162, 2014.
  * UptimeInstitue (2019) UptimeInstitue.  2019 data center industry survey results.  https://uptimeinstitute.com/2019-data-center-industry-survey-results, 2019\.
  * Warden (2018) P. Warden.  Speech commands: A dataset for limited-vocabulary speech recognition.  _arXiv preprint arXiv:1804.03209_ , 2018.
  * Yurochkin et al. (2019) M. Yurochkin, M. Agarwal, S. Ghosh, K. Greenewald, N. Hoang, and Y. Khazaeni.  Bayesian nonparametric federated learning of neural networks.  In _International Conference on Machine Learning_ , pages 7252–7261. PMLR, 2019.
  * Zhang et al. (2017) Y. Zhang, N. Suda, L. Lai, and V. Chandra.  Hello edge: Keyword spotting on microcontrollers.  _arXiv preprint arXiv:1711.07128_ , 2017.