# Fair Evaluation of Federated Learning Algorithms for Automated Breast
Density Classification: The Results of the 2022 ACR-NCI-NVIDIA Federated
Learning Challenge

Kendall Schmidt  Benjamin Bearce  Ken Chang  Laura Coombs  Keyvan Farahani
Marawan Elbatel  Kaouther Mouheb  Robert Marti  Ruipeng Zhang  Yao Zhang
Yanfeng Wang  Yaojun Hu  Haochao Ying  Yuyang Xu  Conrad Testagrose  Mutlu
Demirer  Vikash Gupta  Ünal Akünal  Markus Bujotzek  Klaus H. Maier-Hein  Yi
Qin  Xiaomeng Li  Jayashree Kalpathy-Cramer  Holger R. Roth  American College
of Radiology, USA  The Massachusetts General Hospital, USA  University of
Colorado, USA  National Institutes of Health National Cancer Institute, USA
Computer Vision and Robotics Institute, University of Girona, Spain
Cooperative Medianet Innovation Center, Shanghai Jiao Tong University, China
Shanghai AI Laboratory, China  Real Doctor AI Research Centre, Zhejiang
University, China  School of Public Health, Zhejiang University, China
College of Computer Science and Technology, Zhejiang University, China
University of North Florida College of Computing Jacksonville, USA  Mayo
Clinic Florida Radiology, USA  Division of Medical Image Computing, German
Cancer Research Center, Heidelberg, Germany  Electronic and Computer
Engineering, Hong Kong University of Science and Technology, China  NVIDIA,
USA

###### Abstract

The correct interpretation of breast density is important in the assessment of
breast cancer risk. AI has been shown capable of accurately predicting breast
density, however, due to the differences in imaging characteristics across
mammography systems, models built using data from one system do not generalize
well to other systems. Though federated learning (FL) has emerged as a way to
improve the generalizability of AI without the need to share data, the best
way to preserve features from all training data during FL is an active area of
research. To explore FL methodology, the breast density classification FL
challenge was hosted in partnership with the American College of Radiology,
Harvard Medical Schools’ Mass General Brigham, University of Colorado, NVIDIA,
and the National Institutes of Health National Cancer Institute. Challenge
participants were able to submit docker containers capable of implementing FL
on three simulated medical facilities, each containing a unique large
mammography dataset. The breast density FL challenge ran from June 15 to
September 5, 2022, attracting seven finalists from around the world. The
winning FL submission reached a linear kappa score of 0.653 on the challenge
test data and 0.413 on an external testing dataset, scoring comparably to a
model trained on the same data in a central location.

###### keywords:

Mammography, Breast Density, Federated Learning, Challenge

††journal: Medical Image Analysis

##  1 Introduction

Accurate assignment of breast density from screening mammograms is important
for assessing breast cancer risk and future screening needs of patients as
dense breasts could more easily mask tumors [8]. However, this task is
associated with high inter-rater variability [31]. Though AI models have been
shown to learn to classify breast density with accuracy levels as high as
radiologists [4, 17], mammograms generated from imaging systems or patient
populations outside of those used for AI training can lead to poor
performance. This is especially crucial as imaging technology continues to
evolve and models are deployed in regions beyond the large academic centers
where deep learning models are typically developed [13].

Due to the sensitive nature of medical data, federated learning (FL) across a
distributed data network has emerged as a popular approach to expand the
diversity of training data in hopes of decreasing bias and enhancing
generalizability [28]. During FL, local AI training is performed at multiple
facilities allowing data to stay on-site. Only information about model updates
leave facilities and are combined to create a single federated model.

FL models have been shown to be more generalizable than models trained on
single institution data in medical use cases such as breast density
classification [29], COVID-19 treatment prediction [7], and glioblastoma
segmentation [24]. Despite the promise FL holds for medical AI, the best
approaches for model training (e.g., the use of batch vs. group vs. instance
normalization), model aggregation (e.g., FedAvg, FedProx etc) and workflow
(e.g. federated averaging vs. cyclical weight transfer vs. split learning) are
active areas of research [11].

The recent Federated Tumor Segmentation (FeTS) FL challenge focused on
optimizing aggregation method by holding the model architecture and FL
workflow constant across submissions [25]. The ACR-NCI-NVIDIA breast density
federated learning challenge was proposed to allow participants to explore all
aspects of FL methodology without constraints while also allowing participants
to implement FL algorithms that work in a real distributed environment. Here,
we present on the final algorithms of the challenge participants and
implications for the future of medical FL. Further, we compare the performance
of the FL challenge generated models with a model trained on the same clinical
trial data in a central location on an external dataset and analyze their
performance on different ethnicities and age groups.

(a) Almost entirely fat

(b) Scattered fibroglandular densities

(c) Heterogeneously dense

(d) Extremely dense

Fig. 1: Mammography examples representing each of the four BIRADS breast
density categories.

##  2 Methods

###  2.1 Challenge Overview

The breast density FL challenge was hosted in a siloed environment in the
American College of Radiology (ACR) Azure cloud. Interested participants could
view all challenge rules and request to be registered from the publicly facing
challenge website111https://breastdensityfl.acr.org. After approval from a
challenge admin, participants were given access to a private challenge URL
through which challenge submissions could be made. Participants were tasked
with developing FL algorithms which could accurately classify breast density,
examples in (Fig. 1). Participants were allowed to use pretrained starting
models. Organizers did not participate in the challenge.

Submissions for this challenge were zip archives consisting of instructions to
build docker containers which could launch automated FL runs. The challenge
was broken up into two phases. Phase I, which ran from June 15th 2022 to
September 5th 2022, was for training and developing the FL algorithms. During
Phase I, participants were limited to three submissions per day. Phase II
consisted of running each participant’s last Phase I submission on a hold-out
test dataset. At the end of Phase I, all active participants were given a
questionnaire on their demographics, affiliations, and FL methodology. The
results of the challenge were announced at MICCAI 2022, Singapore with the
winning submission receiving a GPU prize from NVIDIA.

###  2.2 Challenge Architecture

The Challenge architecture consisted of five virtual machines (Fig. 2). One
ran Medical Imaging Challenge Infrastructure (MedICI) software to receive and
distribute challenge submissions. One served as an FL server to coordinate FL
runs across clients. The remaining three were FL clients each with unique
large breast density datasets for model training.

Fig. 2: Breast density federated learning challenge architecture diagram:
Depicted inside the white ACR Azure square are 5 smaller white VM squares.
These are from left to right the MedICI Server, FL Server, and 3 FL clients.
User containers are in red. Other resources include Azure storage (bottom
yellow), Azure docker image container registry (top yellow), data indicated by
“SH data” purple squares, uploaded algorithms as .zip in green. Finally a
firewall was used to control access. After approval by challenge admin, users
were allowed access to submit their FL algorithms through MedICI, which
coordinated the kickoff of FL runs across the ACR FL server and the three ACR
Clients.

####  2.2.1 MedICI

Challenge participants interfaced with the challenge environment via MedICI, a
CodaLab-based [26] open source web platform designed to help data scientists
and research teams to crowd-source the resolution of machine learning problems
through the organization of competitions. MedICI integrates the job scheduling
feature from CodaLab with support for running docker containers built from
uploaded algorithms and code. MedICI was developed at the Massachusetts
General Hospital (MGH) with support from the National Cancer Institute.

An instance of the MedICI website was hosted on the MedICI server (Fig. 2).
Through the MedICI website, participants could submit zip files consisting of
instructions to build their docker images. For security reasons, the automated
docker build process was turned manual in order to review the uploaded docker
build instructions to ensure no malicious content. After review, docker images
were built on the MedICI server then manually pushed to a container
repository. Once pushed, the docker images became available on the platform.

Participants could select to submit FL algorithms contained within their
docker images from a dropdown list at which point MedICI would add the
submission to a queue. MedICI kicked off one submission at a time by
initiating a docker pull and subsequent docker run of the selected container
on the FL components (FL server and 3 FL clients). On completion of the
submission or after 8 hours of runtime, MedICI terminated all running
containers, initiated a push of all generated results from the FL server to a
storage account, and kicked off the next submission in the queue. Results for
each run were stored and displayed to users.

####  2.2.2 FL environment

An FL server coordinated FL submissions across three client machines. Each
client machine had a NVIDIA Tesla V100 GPU and a unique large dataset,
described in more detail below. Though some challenges have restricted
computational resources in order to force computationally efficient
submissions, that was not done here. The full use of each GPU was available to
participants to allow a focus on FL methodology. For a participant submission
to be run successfully, it needed to be fully automated after docker run and
complete within 8 hours of runtime. The total run time was not a factor in
submission ranking. To demonstrate one such submission, a reference submission
using NVIDIA FLARE [30] and MONAI [3] was created and made publicly
available222https://github.com/Project-
MONAI/tutorials/tree/main/federated_learning/breast_density_challenge. Though
not a submission requirement, all submissions throughout the challenge
duration used NVIDIA FLARE.

###  2.3 Challenge Data

Table 1: Number of images per data split across clients.  | Site-1 | Site-2 | Site-3 | Overall
---|---|---|---|---
Training | 22964 | 6534 | 40031 | 69529
Test 1 | 3327 | 1128 | 5952 | 10407
Test 2 | 6637 | 3889 | 13428 | 23954
Total | 32928 | 11551 | 59411 | 103890

Data used for this challenge were obtained as part of the Digital Mammographic
Imaging Screening Trial (DMIST) [27]. ACR holds the IRB for the DMIST dataset
and access must be explicitly approved. Thus, no DMIST data was shared with
challenge participants. As such, a sample dataset was provided along with the
reference example. The sample dataset was based on the Curated Breast Imaging
Subset of Digital Database for Screening Mammography (CBIS-DDSM) dataset [16]
hosted by The Cancer Imaging Archive (TCIA) [6]. Both the DMIST dataset and
the sample dataset were preprocessed as described in [29].

DMIST contains data from 33 institutions and was first split across FL clients
by digital mammography system as described here: [4]. Then, all but the
majority scanner type for each client split were omitted as here [9] and all
patients with data in multiple splits were moved to the test 2 split. Table 1
summarizes the final data splits across clients for the two challenge phases.
As part of DMIST, these data were labeled with one of the four ACR BI-RADS
breast density categories (almost entirely fatty, scattered fibroglandular
densities, heterogeneously dense, or extremely dense). Figure 3 shows the
distribution of BI-RADS scores across sites. Only the labels for the training
data were provided to the participants. Test 1 data was used for Phase I
ranking and test 2 data was used for Phase II ranking.

Fig. 3: BI-RADS scores across sites. 1: Almost entirely fatty, 2: Scattered
areas of fibroglandular tissue, 3: Heterogeneously dense, 4: Extremely dense.

###  2.4 Ranking

Site-level (linear kappa, quadratic kappa, and area under the ROC curve (AUC))
and image-level (linear and quadratic distances) metrics were calculated per
site and overall. These metrics are commonly used to assess breast density
classification AI. While the site-level metrics provided insight on the
overall accuracy of the AI, the image-level metrics provided a measure of the
degree correctness as breast density is measured as an ordinal scale. To rank
each submission, metrics were ranked individually and then averaged to obtain
a single consensus ranking. Given

  * $M$: the number of metrics used for ranking.

  * $R_{ij}$: the rank of participant $i$ for metric $j$.

The average rank $\bar{R_{i}}$ for each participant $i$ was calculated:

$\bar{R_{i}}=\frac{1}{M}\sum_{j=1}^{M}R_{ij}$ (1)

Ranking in this way allowed overall submission accuracy and degree of
correctness to be taken into account. Image-level metrics were used to perform
statistical stability analysis on Phase II rankings with
ChallengeR333https://github.com/wiesenfa/challengeR [33].

##  3 Post-Challenge Analysis

###  3.1 External Validation

All Phase II algorithms were validated on an external test set from MGH. This
evaluation was performed outside the MedICI platform on compute resources at
MGH. The dataset was obtained through IRB approval with patients who had prior
surgery or implants excluded. Each image was read by a breast imaging
radiologist as part of routine clinical practice. All mammograms were acquired
using a Lorad Selenia mammography system (Hologic). The final MGH patient
cohort consisted of 8,603 digital screening images from 1,856 patients. Since
all participants used NVIDIA FLARE, a custom script was created to call the
Learner class for each method and run inference on this novel dataset. Linear
kappa, quadratic kappa and AUC metrics were compared between the Phase II
results and the external validation.

Previously, a model had been trained on DMIST data in a non-federated manner
using similar data splits [4]. We refer to this model as the central DMIST
model. The central DMIST model was also evaluated against the external
validation dataset. Linear distance and quadratic distance were calculated per
image and compared to the FL methods. Since, for the challenge, patients with
data overlap across multiple splits were moved to the test 2 split, the test 2
split contained some data used to train the central DMIST model. Therefore,
this model was not evaluated on the test 2 split.

###  3.2 Demographic Bias Analysis

Table 2: Patient race and age distribution across the different testing sites.  Race | Site-1 | Site-2 | Site-3 | Overall
---|---|---|---|---
American Indian or Alaska | 0 | 0 | 2 | 2
Asian | 33 | 13 | 37 | 83
Black or African American | 176 | 109 | 389 | 674
Hispanic or Latino | 41 | 62 | 72 | 175
Other | 10 | 15 | 18 | 43
Unknown | 5 | 0 | 0 | 5
White | 1122 | 563 | 2237 | 3922
Total | 1387 | 762 | 2755 | 4904
|  |  |  |
Age | Site-1 | Site-2 | Site-3 | Overall
Mean +- Std. Dev. | 55.2+-10.7 | 53.1+-9.8 | 54.2+-10.2 | 54.3+-10.3
Range | 23-90 | 29-89 | 24-88 | 23-90

To assess how different patient demographics can influence the performance of
the finalist methods on the Phase 2 test dataset, for each submission, per-
image metrics were examined along race and age (rounded to the nearest 10
years) categories. Since DMIST data consists of all female subjects, gender
bias was not a concern. Table 2 summarizes the patient self-identified race
and age data across the different sites.

##  4 Results

###  4.1 Phase II submissions

Table 3: Phase II submission summary.  Rank |  Algo. |  Team |  Country |  Base model architecture |  General-Purpose pre-trained weights |  Task-specific pre-trained weights |  FL algorithm
---|---|---|---|---|---|---|---
1 |  #1 |  Elbatel, Mouheb, Marti |  Spain |  ResNet50 |  No |  Yes |  FedProx
2 |  #2 |  Zhang, Zhang, Wang |  China |  ResNet50 |  Yes |  No |  FedAvg
3 |  #3 |  Hu, Ying, Xu |  China |  DenseNet-121 |  Yes |  No |  SCAFFOLD
4 |  #4 |  Testagrose, Demirer, Gupta, Liu, Erdal, White |  USA |  Inception-V3 |  Yes |  Yes |  FedAvg
5 |  #6 |  Akünal, Bujotzek, Maier-Hein |  Germany |  ResNet18 |  Yes |  No |  FedAvg
6 |  #5 |  Bahrini |  Tunesia |  ResNet50 |  Yes |  No |  FedAvg
7 |  #7 |  Qin, Li |  China |  ResNet56 |  No |  No |  FedAvg

A summary of the Phase II submissions based on the participant questionnaire
and provided abstracts is shown in Table 3. For each submission, a brief
summary is provided below, with more detailed summaries available in A. Note
these summaries were provided by the participants. Challenge organizers had
limited insight into the methods beyond what was provided by participants.

####  4.1.1 1st Rank (Algo. #1)

This method began with normalizing the imaging data histograms. A ResNet50
architecture pre-trained on task-specific data was used along with a rank-
consistent layer for ordinal classification. For 100 rounds, FedProx [19] was
used after 1 epoch. The classification layer was not globally aggregated, thus
each site maintained a locally fine-tuned classification layer. For external
validation, the three local models generated during FL were treated as an
ensemble.

####  4.1.2 2nd Rank (Algo. #2)

This method used a ResNet50 architecture pretrained on generic image data.
Like the 1st rank algorithm, this method also maintained local classifiers at
each site, however, a global classifier was also maintained. Model aggregation
occurred via FedAvg [20] for 80 rounds each after two local training epochs.

####  4.1.3 3rd Rank (Algo. #3)

A DenseNet121 architecture was used in this method with SCAFFOLD [12] for
model aggregation. This method also employed Top-k sparsification perhaps due
to communication bandwidth issues experienced with prior submissions.

####  4.1.4 4th Rank (Algo. #4)

For this method the Inception-V3 [32] architecture pre-trained on generic
image data was used. Prior to submitting to the challenge, the model was fine-
tuned on task specific data from the developer’s facility. The method employed
FedAvg for model aggregation.

####  4.1.5 5th Rank (Algo. #6)

A ResNet18 [10] model architecture was used with FedAvg for aggregation after
1 epoch for 100 rounds. FedProx was also tested in this method but ultimately
found to be less accurate than FedAvg. Other model architectures were tried,
but issues were faced in terms of finishing prior to the 8 hour time limit or
with communication bandwidth.

####  4.1.6 6th Rank (Algo. #5)

This participant did not provide a description of this method.

####  4.1.7 7th Rank (Algo. #7)

A ResNet56 model was used for this method. Aggregation occurred via FedAvg,
weighted by the proportion of total data points from each site. Between each
round, each site trained for 5 epochs.

###  4.2 Challenge evaluation

Table 4: Average performance across final algorithms.  |  Lin. Kappa Site-1 |  Quad. Kappa Site-1 |  AUC Site-1 |  Lin. Kappa Site-2 |  Quad. Kappa Site-2 |  AUC Site-2 |  Lin. Kappa Site-3 |  Quad. Kappa Site-3 |  AUC Site-3 |  Lin. Kappa All |  Quad. Kappa All |  AUC All
---|---|---|---|---|---|---|---|---|---|---|---|---
Phase I (Train & Val.) |  0.59 |  0.69 |  0.9 |  0.49 |  0.59 |  0.92 |  0.65 |  0.75 |  0.93 |  0.62 |  0.72 |  0.91
Phase II (Test) |  0.63 |  0.73 |  0.93 |  0.47 |  0.58 |  0.90 |  0.66 |  0.75 |  0.94 |  0.62 |  0.72 |  0.93
Relative Change |  7.42% |  5.79% |  2.81% |  -4.62% |  -1.40% |  -2.24% |  0.64% |  0.55% |  0.85% |  0.93% |  0.90% |  1.28%

The average performance of the final algorithms across evaluation metrics
between Phase I and Phase II is shown in Table 4. On average, the final
algorithms performed slightly better on the Phase II data but show a
relatively stable performance across the two phases of the challenge.

(a)  (b)

(c)

Fig. 4: Site-based metrics in Phase II (Test phase).

Figure 4 shows the average distribution of evaluation metrics across the
different sites. Specifically, the y axis shows the number of final algorithms
to fall into a value range for linear kappa, quadratic kappa, and AUC metrics,
shown on the x axis of Figure 4 a-c respectfully. From the left-most position
of the red bars on all plots, one can observe that the performance of the
algorithms is markedly lower on site 2, which included the least training data
(as shown in Table 1).

We used the statistical ranking toolkit proposed by Wiesenfarth et al. [33] to
analyze the stability of the ranking results. For this evaluation, the
algorithms were randomly numbered from #1 to #7 and evaluated using the per-
image metrics.

(a) Overall Linear Distance

(b) Overall Quadratic Distance

Fig. 5: Ranking stability using per-image distance metrics in Phase II (Test
phase).

In Fig. 5, we investigated the ranking stability of the algorithms using the
overall per-image metrics (linear and quadratic). The size of each marker at a
position represents how often a specific algorithm achieved a certain rank out
of 1000 trials using bootstrapping. The black cross shows the median rank for
each algorithm, and the black lines show the 95% range of values from all
trials.

(a) Overall Linear Distance

(b) Overall Quadratic Distance

Fig. 6: Significance maps for overall per-image distance metrics in Phase II
(Test phase).

Finally, we compare the statistical significance with which algorithms
outperformed other algorithms in different per-image metric tasks. Figure 6
shows the significance maps on the overall linear and quadratic per-image
metric tasks. Yellow shading indicates the algorithm on the x-axis had
significantly better performance than the one on the y-axis. Blue shading
indicates no significant difference between the two algorithms.

Additional insight into the assessment data utilized in the ranking is given
in B.

###  4.3 External evaluation

Table 5: Comparison of final challenge test ranking and external validation
ranking.

Rank |  Algo. |  Avg. Rank |  Lin. Kappa Site-1 |  Quad. Kappa Site-1 |  AUC Site-1 |  Lin. Kappa Site-2 |  Quad. Kappa Site-2 |  AUC Site-2 |  Lin. Kappa Site-3 |  Quad. Kappa Site-3 |  AUC Site-3 |  Lin. Kappa all |  Quad. Kappa all |  AUC all
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
1st |  #1 |  1.45 |  0.667 |  0.767 |  0.934 |  0.574 |  0.678 |  0.908 |  0.665 |  0.761 |  0.936 |  0.653 |  0.752 |  0.930
2nd |  #2 |  1.75 |  0.658 |  0.749 |  0.933 |  0.498 |  0.605 |  0.911 |  0.671 |  0.764 |  0.936 |  0.643 |  0.739 |  0.927
3rd |  #3 |  3.15 |  0.605 |  0.711 |  0.929 |  0.493 |  0.607 |  0.905 |  0.671 |  0.766 |  0.939 |  0.626 |  0.729 |  0.928
4th |  #4 |  3.80 |  0.621 |  0.721 |  0.931 |  0.476 |  0.589 |  0.908 |  0.655 |  0.757 |  0.935 |  0.620 |  0.725 |  0.925
5th |  #6 |  4.80 |  0.614 |  0.716 |  0.930 |  0.419 |  0.535 |  0.897 |  0.647 |  0.738 |  0.935 |  0.604 |  0.704 |  0.925
6th |  #5 |  4.85 |  0.627 |  0.728 |  0.924 |  0.421 |  0.533 |  0.894 |  0.646 |  0.747 |  0.938 |  0.608 |  0.713 |  0.922
7th |  #7 |  5.65 |  0.623 |  0.722 |  0.928 |  0.420 |  0.532 |  0.901 |  0.639 |  0.743 |  0.934 |  0.603 |  0.710 |  0.924
(a) Final Test Phase Ranking (Phase II)

Rank |  Algo. |  Avg. Rank |  Lin. Kappa |  Quad. Kappa |  AUC
---|---|---|---|---|---
1st |  #1 |  1.8 |  0.413 |  0.555 |  0.917
2nd |  #2 |  2.0 |  0.357 |  0.423 |  0.860
3rd |  #3 |  3.0 |  0.350 |  0.427 |  0.851
4th |  #4 |  4.0 |  0.243 |  0.354 |  0.880
5th |  #7 |  4.2 |  0.218 |  0.273 |  0.855
6th |  #6 |  5.4 |  0.172 |  0.253 |  0.853
7th |  #5 |  6.2 |  0.116 |  0.161 |  0.779
Central |  |  N/A |  0.468 |  0.600 |  0.924
(b) External Validation Ranking

After the challenge concluded, we evaluated the finalist algorithms on an
external testing dataset that was not used during the training and test phases
of the challenge. Here, we compare the consistency of the algorithm ranking on
this external dataset with the Phase II challenge ranking. Table 5 shows both
the final ranking of the challenge, and the ranking for the same algorithms on
the external validation set. While one can observe a noticeable drop in the
Kappa and AUC metrics for the external validation, the ranking is relatively
consistent with the challenge test ranking. The central DMIST model was also
evaluated against the same external testing dataset and results are shown as
the last row in Table 5.

###  4.4 Demographic Analysis

Fig. 7: Boxplots showing the quadratic Kappa performance distributions of all
finalist algorithms on the Phase II test set.  Fig. 8: Average quadratic kappa
performance of the finalists on the Phase II test set. The shaded area shows
the 95% confidence intervals.

The box plots in Fig. 7 show the distribution of the finalists algorithms
across race. Finally, we compared the average performance of the different
finalist algorithms across the age of the patients (rounded to the closest 10)
in Fig. 8.

##  5 Discussion

The breast density FL challenge attracted world-wide participation and yielded
many insights into FL methodology. It also represents a framework for hosting
future AI challenges in which data is not distributed to participants and
remains behind firewalls. Though conducting the challenge in this way likely
required more hands-on debugging from challenge organizers, it made possible
the utilization of a rich clinical trial dataset which would ordinarily
require explicit IRB approval.

All final FL algorithms showed worse performance on site 2, the site with the
smallest dataset, than the other two sites. This indicates the global models
had better learned features from the more prevalent data types. Interestingly,
the use of state-of-the-art FL algorithms such as FedProx [19] and SCAFFOLD
[12] did not guarantee higher rank on image-level metrics from site 2. Still,
FedProx was used in the rank 1 algorithm which consistently ranked 1st on site
2 image-level metrics. This can likely be attributed to the reduced model
drift when using FedProx as the algorithm introduces an additional penalty
term to avoid the models moving to far away from the current global model
during local training [19]. However, during Phase I, the rank 5 team found
that using FedProx in their implementation resulted in lower overall
performance than FedAvg and opted to use FedAvg in Phase II. This might have
been caused by a sub-optimal hyperparameter choice. The rank 2 and 3
algorithms had similar performance on site 2 data, but the rank 2 algorithm
used FedAvg while the rank 3 algorithm used SCAFFOLD. SCAFFOLD adds control
terms to the local model updates to correct for client drift [12]. Given these
findings, in this challenge using a real-world data distribution, the
aggregation method perhaps proved less relevant to the overall algorithm
performances than other methodological choices. We suspect that pre-training
and model ensembling played a bigger role in improving the overall robustness
of the algorithms.

While most of the finalist algorithms began FL runs with pretrained model
weights, the rank 7 algorithm reported using randomly initialized weights.
Using pretrained weights in FL has been shown to enhance performance of global
models [22]. Of the finalist algorithms which used pretrained weights, ranks
2,3,5 and 6 used general purpose pretrained weights. The rank 1 and 4
algorithms used task specific external data for model pretraining. Task
specific pre-training may have contributed to the overall performance of these
algorithms, but alone did not guarantee a top rank.

The rank 1 and rank 2 algorithms showed similar performance across all
metrics, though their approaches were very different. While both algorithms
maintained local classifiers at each site, the rank 1 algorithm relied
exclusively on the local classifiers for making predictions while the rank 2
algorithm used the local classifiers to constrain updates to a global
classifier. Thus, the rank 1 submission did not generate a single complete
global model. For validation against the external dataset for that algorithm,
the three local site models were treated as an ensemble and the logits were
averaged to obtain a single set of probabilities. From there, metrics were
computed as per all other submissions. Though it is well known that local fine
tuning often improves AI performance, the lack of a global classifier could
pose logistical challenges for AI model distribution. However, in the case of
the rank 1 algorithm, the external validation performance was still highest
relative to the other challenge algorithms.

As all but one finalist algorithm produced a single global model, we
considered whether ensembling models could be considered unfair. However, the
challenge rules did not specifically disallow ensembles of models.
Additionally, the stated purpose of this challenge was to allow participants
to explore all aspects of FL methodology without constraints, the goal being
to find the approach that works best for this FL task. Hence, it was
determined that there should be no restrictions against model ensembles.

Our evaluation of the finalist algorithms on an external testing dataset
confirmed a consistent ranking of the top algorithms. This consistency
confirms the winning algorithm also generalizes best to this external test
set, albeit with a performance drop. Indeed, the ranking order for the first
four algorithms stays the same. Only the ranking in 5th to 7th rank changed on
this external dataset, further validating the robustness of our ranking
results for the challenge. Though the central DMIST model achieved the highest
overall performance on the external testing dataset, the central metrics are
similar to the rank 1 FL algorithm.

Aside from capping runtime at 8 hours, computational efficiency was not
tracked in this challenge. Unlike some challenges where computational
resources are restricted, the participants had full use of all virtual machine
resources. That said, in their final algorithm summaries, some participants
mentioned encountering issues related to exceeding the allotted 8 hour runtime
and lack of sufficient messaging bandwidth between clients and server. During
the challenge, challenge organizers actively reported details around each
submission failure in order to enable participants to alter their submissions
to achieve success. Anecdotally, participants coped with runtime issues by
choosing smaller model architectures, reducing the number of training rounds
or reducing the number of local epochs between training rounds. The
communication bandwidth was imposed at 2 GB by gRPC, the remote procedure call
framework used by NVIDIA FLARE. Participants worked around these issues by
choosing smaller model architectures. The rank 3 algorithm also used Top-k
sparsification to decrease the client-to-server message size444Note, the
latest 2.4.0 version of NVIDIA FLARE now has a streaming layer that allows
message sizes exceeding 2 GB.

The demographic analysis showed that while most algorithms performed
relatively comparably across the different race populations and age groups,
some markedly lower performances in the latino category can be observed, which
is likely attributed to the relatively smaller training data for this race
group. We again see a performance drop for most race classes on site 2
compared to sites 1 and 3. The race classes with largest representation
(White, Black or African American, and Hispanic or Latino) have stable
performance at sites 1 and 3. Some minority race groups like “american indian
or alaska” have too few samples to draw conclusive results, but we included
them for completeness. These findings point to further possibilities for
research into fairness and unbiased AI, which could help address these issues.
Personalization for specific ethnicities might be another opportunity to
address these issues [34].

##  6 Conclusion

In conclusion, the breast density FL challenge provided insights into FL
methodology and a framework for hosting future distributed AI challenges
without access to the data for participants, allowing for a fair comparison
between the algorithms. Importantly, we have demonstrated how FL algorithm
development can be completely decoupled from access to patient data itself, an
important proof-of-concept for medical AI that extends beyond competitions.
Although the FL algorithms generalized well on the challenge testing set, the
drops in performance on external data and minority groups further highlight
the need for research into fairness and unbiased and personalized AI models.
We hope that the challenge’s success encourages hosting future AI challenges
for fair evaluation of state-of-the-art FL methods.

## Acknowledgments

The environment for this challenge was supported by The Microsoft AI for
Health grant. M.E. K.M. are partially funded by the EACEA Erasmus Mundus
grant, R.M. acknowledges the funding from the Spanish Science and Education
Ministry, grant no. PID2021-123390OB-C21. R.Z. is partially supported by Wu
Wen Jun Honorary Doctoral Scholarship, AI Institute, Shanghai Jiao Tong
University.

Funding for challenge support, via a subcontract, was provided by Leidos
Biomedical Research, Inc which provides operational and technical support to
the Frederick National Laboratory for Cancer Research, a Federally Funded
Research and Development Center (FFRDC). Operational and technical support
involves the management and execution of projects sponsored by the National
Cancer Institute (NCI) Center for Biomedical Informatics and Information
Technology (CBIIT). The Medical Imaging Challenge Infrastructure (MedICI)
Platform was developed to support imaging, digital pathology, and genomics
challenges. This system was built on a framework of open-source projects and
was used in the breast density classification challenge.

## References

  * Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton [2017] Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton, 2017\.  ImageNet classification with deep convolutional neural networks.  Communications of the ACM .
  * Cao et al. [2020] Cao, W., Mirjalili, V., Raschka, S., 2020.  Rank consistent ordinal regression for neural networks with application to age estimation.  Pattern Recognition Letters 140, 325–331.
  * Cardoso et al. [2022] Cardoso, M.J., Li, W., Brown, R., Ma, N., Kerfoot, E., Wang, Y., Murray, B., Myronenko, A., Zhao, C., Yang, D., Nath, V., He, Y., Xu, Z., Hatamizadeh, A., Zhu, W., Liu, Y., Zheng, M., Tang, Y., Yang, I., Zephyr, M., Hashemian, B., Alle, S., Zalbagi Darestani, M., Budd, C., Modat, M., Vercauteren, T., Wang, G., Li, Y., Hu, Y., Fu, Y., Gorman, B., Johnson, H., Genereaux, B., Erdal, B.S., Gupta, V., Diaz-Pinto, A., Dourson, A., Maier-Hein, L., Jaeger, P.F., Baumgartner, M., Kalpathy-Cramer, J., Flores, M., Kirby, J., Cooper, L.A., Roth, H.R., Xu, D., Bericat, D., Floca, R., Zhou, S.K., Shuaib, H., Farahani, K., Maier-Hein, K.H., Aylward, S., Dogra, P., Ourselin, S., Feng, A., 2022\.  MONAI: An open-source framework for deep learning in healthcare doi:https://doi.org/10.48550/arXiv.2211.02701.
  * Chang et al. [2020] Chang, K., Beers, A.L., Brink, L., Patel, J.B., Singh, P., Arun, N.T., Hoebel, K.V., Gaw, N., Shah, M., Pisano, E.D., Tilkin, M., Coombs, L.P., Dreyer, K.J., Allen, B., Agarwal, S., Kalpathy-Cramer, J., 2020\.  Multi-Institutional assessment and crowdsourcing evaluation of deep learning for automated classification of breast density.  J. Am. Coll. Radiol. 17, 1653–1662.
  * Chen and Chao [2021] Chen, H.Y., Chao, W.L., 2021\.  On bridging generic and personalized federated learning for image classification arXiv:2107.00778.
  * Clark et al. [2013] Clark, K., Vendt, B., Smith, K., Freymann, J., Kirby, J., Koppel, P., Moore, S., Phillips, S., Maffitt, D., Pringle, M., Tarbox, L., Prior, F., 2013\.  The cancer imaging archive (TCIA): maintaining and operating a public information repository.  J. Digit. Imaging 26, 1045--1057.
  * Dayan et al. [2021] Dayan, I., Roth, H.R., Zhong, A., Harouni, A., Gentili, A., Abidin, A.Z., Liu, A., Costa, A.B., Wood, B.J., Tsai, C.S., et al., 2021\.  Federated learning for predicting clinical outcomes in patients with covid-19.  Nature medicine 27, 1735--1743.
  * Freer [2015] Freer, P.E., 2015.  Mammographic breast density: impact on breast cancer risk and implications for screening.  Radiographics 35, 302--315.
  * Gupta et al. [2021] Gupta, S., Singh, P., Chang, K., Qu, L., Aggarwal, M., Arun, N., Vaswani, A., Raghavan, S., Agarwal, V., Gidwani, M., Hoebel, K., Patel, J., Lu, C., Bridge, C.P., Rubin, D.L., Kalpathy-Cramer, J., 2021\.  Addressing catastrophic forgetting for medical domain expansion .
  * He et al. [2016] He, K., Zhang, X., Ren, S., Sun, J., 2016.  Deep residual learning for image recognition, in: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770--778.
  * Kairouz et al. [2019] Kairouz, P., McMahan, H.B., Avent, B., Bellet, A., Bennis, M., Bhagoji, A.N., Bonawitz, K., Charles, Z., Cormode, G., Cummings, R., D’Oliveira, R.G.L., Eichner, H., Rouayheb, S.E., Evans, D., Gardner, J., Garrett, Z., Gascón, A., Ghazi, B., Gibbons, P.B., Gruteser, M., Harchaoui, Z., He, C., He, L., Huo, Z., Hutchinson, B., Hsu, J., Jaggi, M., Javidi, T., Joshi, G., Khodak, M., Konečný, J., Korolova, A., Koushanfar, F., Koyejo, S., Lepoint, T., Liu, Y., Mittal, P., Mohri, M., Nock, R., Özgür, A., Pagh, R., Raykova, M., Qi, H., Ramage, D., Raskar, R., Song, D., Song, W., Stich, S.U., Sun, Z., Suresh, A.T., Tramèr, F., Vepakomma, P., Wang, J., Xiong, L., Xu, Z., Yang, Q., Yu, F.X., Yu, H., Zhao, S., 2019.  Advances and open problems in federated learning arXiv:1912.04977.
  * Karimireddy et al. [2019] Karimireddy, S.P., Kale, S., Mohri, M., Reddi, S.J., Stich, S.U., Suresh, A.T., 2019\.  SCAFFOLD: Stochastic controlled averaging for federated learning .
  * Kaushal et al. [2020] Kaushal, A., Altman, R., Langlotz, C., 2020.  Geographic distribution of US cohorts used to train deep learning algorithms.  JAMA 324, 1212--1213.
  * Kingma and Ba [2014] Kingma, D.P., Ba, J., 2014.  Adam: A method for stochastic optimization arXiv:1412.6980.
  * Konz et al. [2023] Konz, N., Buda, M., Gu, H., Saha, A., Yang, J., Chledowski, J., Park, J., Witowski, J., Geras, K.J., Shoshan, Y., Gilboa-Solomon, F., Khapun, D., Ratner, V., Barkan, E., Ozery-Flato, M., Martí, R., Omigbodun, A., Marasinou, C., Nakhaei, N., Hsu, W., Sahu, P., Hossain, M.B., Lee, J., Santos, C., Przelaskowski, A., Kalpathy-Cramer, J., Bearce, B., Cha, K., Farahani, K., Petrick, N., Hadjiiski, L., Drukker, K., Armato, 3rd, S.G., Mazurowski, M.A., 2023.  A competition, benchmark, code, and data for using artificial intelligence to detect lesions in digital breast tomosynthesis.  JAMA Netw Open 6, e230524.
  * Lee et al. [2017] Lee, R.S., Gimenez, F., Hoogi, A., Miyake, K.K., Gorovoy, M., Rubin, D.L., 2017\.  A curated mammography data set for use in computer-aided detection and diagnosis research.  Sci Data 4, 170177\.
  * Lehman et al. [2019] Lehman, C.D., Yala, A., Schuster, T., Dontchos, B., Bahl, M., Swanson, K., Barzilay, R., 2019.  Mammographic breast density assessment using deep learning: Clinical implementation.  Radiology 290, 52--58.
  * Li et al. [2021] Li, Q., He, B., Song, D., 2021\.  Model-Contrastive federated learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10713--10722.
  * Li et al. [2018] Li, T., Sahu, A.K., Zaheer, M., Sanjabi, M., Talwalkar, A., Smith, V., 2018\.  Federated optimization in heterogeneous networks .
  * McMahan et al. [2017] McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A., 2017.  Communication-Efficient Learning of Deep Networks from Decentralized Data, PMLR, Fort Lauderdale, FL, USA. pp. 1273--1282.
  * Mendieta et al. [2022] Mendieta, M., Yang, T., Wang, P., Lee, M., Ding, Z., Chen, C., 2022.  Local learning matters: Rethinking data heterogeneity in federated learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8397--8406.
  * Nguyen et al. [2022] Nguyen, J., Wang, J., Malik, K., Sanjabi, M., Rabbat, M., 2022.  Where to begin? on the impact of pre-training and initialization in federated learning .
  * Paszke et al. [2019] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S., 2019.  PyTorch: An imperative style, High-Performance deep learning library.  Adv. Neural Inf. Process. Syst. 32\.
  * Pati et al. [2023] Pati, S., Baid, U., Edwards, B., Sheller, M., Wang, S.H., Reina, G.A., Foley, P., Gruzdev, A., Karkada, D., Davatzikos, C., Sako, C., Ghodasara, S., Bilello, M., Mohan, S., Vollmuth, P., Brugnara, G., Preetha, C.J., Sahm, F., Maier-Hein, K., Zenk, M., Bendszus, M., Wick, W., Calabrese, E., Rudie, J., Villanueva-Meyer, J., Cha, S., Ingalhalikar, M., Jadhav, M., Pandey, U., Saini, J., Garrett, J., Larson, M., Jeraj, R., Currie, S., Frood, R., Fatania, K., Huang, R.Y., Chang, K., Balaña, C., Capellades, J., Puig, J., Trenkler, J., Pichler, J., Necker, G., Haunschmidt, A., Meckel, S., Shukla, G., Liem, S., Alexander, G.S., Lombardo, J., Palmer, J.D., Flanders, A.E., Dicker, A.P., Sair, H.I., Jones, C.K., Venkataraman, A., Jiang, M., So, T.Y., Chen, C., Heng, P.A., Dou, Q., Kozubek, M., Lux, F., Michálek, J., Matula, P., Keřkovský, M., Kopřivová, T., Dostál, M., Vybíhal, V., Vogelbaum, M.A., Mitchell, J.R., Farinhas, J., Maldjian, J.A., Yogananda, C.G.B., Pinho, M.C., Reddy, D., Holcomb, J., Wagner, B.C., Ellingson, B.M., Cloughesy, T.F., Raymond, C., Oughourlian, T., Hagiwara, A., Wang, C., To, M.S., Bhardwaj, S., Chong, C., Agzarian, M., Falcão, A.X., Martins, S.B., Teixeira, B.C.A., Sprenger, F., Menotti, D., Lucio, D.R., LaMontagne, P., Marcus, D., Wiestler, B., Kofler, F., Ezhov, I., Metz, M., Jain, R., Lee, M., Lui, Y.W., McKinley, R., Slotboom, J., Radojewski, P., Meier, R., Wiest, R., Murcia, D., Fu, E., Haas, R., Thompson, J., Ormond, D.R., Badve, C., Sloan, A.E., Vadmal, V., Waite, K., Colen, R.R., Pei, L., Ak, M., Srinivasan, A., Bapuraj, J.R., Rao, A., Wang, N., Yoshiaki, O., Moritani, T., Turk, S., Lee, J., Prabhudesai, S., Morón, F., Mandel, J., Kamnitsas, K., Glocker, B., Dixon, L.V.M., Williams, M., Zampakis, P., Panagiotopoulos, V., Tsiganos, P., Alexiou, S., Haliassos, I., Zacharaki, E.I., Moustakas, K., Kalogeropoulou, C., Kardamakis, D.M., Choi, Y.S., Lee, S.K., Chang, J.H., Ahn, S.S., Luo, B., Poisson, L., Wen, N., Tiwari, P., Verma, R., Bareja, R., Yadav, I., Chen, J., Kumar, N., Smits, M., van der Voort, S.R., Alafandi, A., Incekara, F., Wijnenga, M.M.J., Kapsas, G., Gahrmann, R., Schouten, J.W., Dubbink, H.J., Vincent, A.J.P.E., van den Bent, M.J., French, P.J., Klein, S., Yuan, Y., Sharma, S., Tseng, T.C., Adabi, S., Niclou, S.P., Keunen, O., Hau, A.C., Vallières, M., Fortin, D., Lepage, M., Landman, B., Ramadass, K., Xu, K., Chotai, S., Chambless, L.B., Mistry, A., Thompson, R.C., Gusev, Y., Bhuvaneshwar, K., Sayah, A., Bencheqroun, C., Belouali, A., Madhavan, S., Booth, T.C., Chelliah, A., Modat, M., Shuaib, H., Dragos, C., Abayazeed, A., Kolodziej, K., Hill, M., Abbassy, A., Gamal, S., Mekhaimar, M., Qayati, M., Reyes, M., Park, J.E., Yun, J., Kim, H.S., Mahajan, A., Muzi, M., Benson, S., Beets-Tan, R.G.H., Teuwen, J., Herrera-Trujillo, A., Trujillo, M., Escobar, W., Abello, A., Bernal, J., Gómez, J., Choi, J., Baek, S., Kim, Y., Ismael, H., Allen, B., Buatti, J.M., Kotrotsou, A., Li, H., Weiss, T., Weller, M., Bink, A., Pouymayou, B., Shaykh, H.F., Saltz, J., Prasanna, P., Shrestha, S., Mani, K.M., Payne, D., Kurc, T., Pelaez, E., Franco-Maldonado, H., Loayza, F., Quevedo, S., Guevara, P., Torche, E., Mendoza, C., Vera, F., Ríos, E., López, E., Velastin, S.A., Ogbole, G., Soneye, M., Oyekunle, D., Odafe-Oyibotha, O., Osobu, B., Shu’aibu, M., Dorcas, A., Dako, F., Simpson, A.L., Hamghalam, M., Peoples, J.J., Hu, R., Tran, A., Cutler, D., Moraes, F.Y., Boss, M.A., Gimpel, J., Veettil, D.K., Schmidt, K., Bialecki, B., Marella, S., Price, C., Cimino, L., Apgar, C., Shah, P., Menze, B., Barnholtz-Sloan, J.S., Martin, J., Bakas, S., 2023\.  Author correction: Federated learning enables big data for rare cancer boundary detection.  Nat. Commun. 14, 436\.
  * Pati et al. [2021] Pati, S., Baid, U., Zenk, M., Edwards, B., Sheller, M., Reina, G.A., Foley, P., Gruzdev, A., Martin, J., Albarqouni, S., Chen, Y., Shinohara, R.T., Reinke, A., Zimmerer, D., Freymann, J.B., Kirby, J.S., Davatzikos, C., Colen, R.R., Kotrotsou, A., Marcus, D., Milchenko, M., Nazeri, A., Fathallah-Shaykh, H., Wiest, R., Jakab, A., Weber, M.A., Mahajan, A., Maier-Hein, L., Kleesiek, J., Menze, B., Maier-Hein, K., Bakas, S., 2021.  The federated tumor segmentation (FeTS) challenge .
  * Pavao et al. [2022] Pavao, A., Guyon, I., Letournel, A.C., Baró, X., Escalante, H., Escalera, S., Thomas, T., Xu, Z., 2022.  CodaLab Competitions: An open source platform to organize scientific challenges.  Ph.D. thesis. Université Paris-Saclay, FRA.
  * Pisano et al. [2005] Pisano, E.D., Gatsonis, C., Hendrick, E., Yaffe, M., Baum, J.K., Acharyya, S., Conant, E.F., Fajardo, L.L., Bassett, L., D’Orsi, C., Jong, R., Rebner, M., Digital Mammographic Imaging Screening Trial (DMIST) Investigators Group, 2005.  Diagnostic performance of digital versus film mammography for breast-cancer screening.  N. Engl. J. Med. 353, 1773--1783.
  * Rieke et al. [2020] Rieke, N., Hancox, J., Li, W., Milletarì, F., Roth, H.R., Albarqouni, S., Bakas, S., Galtier, M.N., Landman, B.A., Maier-Hein, K., Ourselin, S., Sheller, M., Summers, R.M., Trask, A., Xu, D., Baust, M., Cardoso, M.J., 2020\.  The future of digital health with federated learning.  npj Digital Medicine 3, 1--7.
  * Roth et al. [2020] Roth, H.R., Chang, K., Singh, P., Neumark, N., Li, W., Gupta, V., Gupta, S., Qu, L., Ihsani, A., Bizzo, B.C., Wen, Y., Buch, V., Shah, M., Kitamura, F., Mendonça, M., Lavor, V., Harouni, A., Compas, C., Tetreault, J., Dogra, P., Cheng, Y., Erdal, S., White, R., Hashemian, B., Schultz, T., Zhang, M., McCarthy, A., Min Yun, B., Sharaf, E., Hoebel, K.V., Patel, J.B., Chen, B., Ko, S., Leibovitz, E., Pisano, E.D., Coombs, L., Xu, D., Dreyer, K.J., Dayan, I., Naidu, R.C., Flores, M., Rubin, D., Kalpathy-Cramer, J., 2020\.  Federated learning for breast density classification: A Real-World implementation.
  * Roth et al. [2022] Roth, H.R., Cheng, Y., Wen, Y., Yang, I., Xu, Z., Hsieh, Y.T., Kersten, K., Harouni, A., Zhao, C., Lu, K., Zhang, Z., Li, W., Myronenko, A., Yang, D., Yang, S., Rieke, N., Quraini, A., Chen, C., Xu, D., Ma, N., Dogra, P., Flores, M., Feng, A., 2022\.  NVIDIA FLARE: Federated learning from simulation to Real-World arXiv:2210.13291.
  * Sprague et al. [2016] Sprague, B.L., Conant, E.F., Onega, T., Garcia, M.P., Beaber, E.F., Herschorn, S.D., Lehman, C.D., Tosteson, A.N.A., Lacson, R., Schnall, M.D., Kontos, D., Haas, J.S., Weaver, D.L., Barlow, W.E., PROSPR Consortium, 2016.  Variation in mammographic breast density assessments among radiologists in clinical practice: A multicenter observational study.  Ann. Intern. Med. 165, 457--464.
  * Szegedy et al. [2016] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z., 2016.  Rethinking the inception architecture for computer vision, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2818--2826.
  * Wiesenfarth et al. [2021] Wiesenfarth, M., Reinke, A., Landman, B.A., Eisenmann, M., Saiz, L.A., Cardoso, M.J., Maier-Hein, L., Kopp-Schneider, A., 2021\.  Author correction: Methods and open-source toolkit for analyzing and visualizing challenge results.  Sci. Rep. 11, 10144\.
  * Zhong et al. [2022] Zhong, A., He, H., Ren, Z., Li, N., Li, Q., 2022\.  FedDAR: Federated Domain-Aware representation learning arXiv:2209.04007.

##  Appendix A Descriptions of Finalist Algorithms

1st Rank (Algo. #1)
Federated Breast Density Estimation with Deep Ordinal Regression
Marawan Elbatel∗, Kaouther Mouheb∗, Robert Marti

Computer Vision and Robotics Institute, University of Girona.
∗Equal contribution

Inspired by the fact that local learning matters in federated learning (FL)
[Chen and Chao, 2021, Mendieta et al., 2022], we focus on local client
training in our submission. First, we pre-process the non-IID input mammograms
across different sites using histogram equalization following a previous
observation [Roth et al., 2020]. Although this equalization can result in a
loss of intensity information, such harmonization ensured consistent learning
among different sites in FL. As the relative ordering between breast density
classes is significant, we propose to leverage such information and formulate
the problem as an ordinal classification task. Thus, we add a rank-consistent
layer [Cao et al., 2020] on top of the feature encoder, ResNet50, to perform
the deep ordinal classification. For faster convergence in the FL setting, we
initialize the local clients’ models with weights extracted from an upstream
object detection task on the DBTeX challenge [Konz et al., 2023]. We utilize
our local training procedure in the NVflare framework with FedAvg [Konz et
al., 2023, McMahan et al., 2017], FedProx [Mendieta et al., 2022, Li et al.,
2018], FedAlignMendieta et al. [2022], and MOON [Li et al., 2021].
Experimental results showed that FedProx yielded the highest performance over
the validation set. Finally, we train all clients with FedProx with one local
epoch for 100 communication rounds.

2nd Rank (Algo. #2)
A Model Mixture Framework for Personalized Federated Learning in Breast
Density Estimation
Ruipeng Zhang1,2, Yao Zhang2, Yanfeng Wang1,2

1Cooperative Medianet Innovation Center, Shanghai Jiao Tong University
2Shanghai AI Laboratory

The Breast Density FL challenge aims at developing generalizable models from
distributed datasets for breast density estimation. While federated learning
methods, e.g., FedAvg, enable learning from multi-site datasets without data
sharing, the heterogenous data still remains a great challenge to current
methods. The data heterogeneity lies in two aspects: the variance of data
distribution and the imbalance of classes among multi-site datasets.
Therefore, we are motivated to avoid the global model being biased by the
dominant site while keeping the local models powerful for their corresponding
sites.

To this end, we propose a personalized federated learning framework based on a
model mixture. In our framework, each site has its own local model that
consists of a shared feature extractor and two classifiers: one local and one
global. Both local and global classifiers learn from the local dataset. In
contrast, only the parameters of the global classifier and those of the
feature extractor are aggregated and updated to form the global model during
the federated learning process. For balanced training, we apply a cross-
entropy loss to both the sum of two classifiers’ predictions and another
solely for only the global classifier.

Specifically, we employ a ResNet50 pre-trained on ImageNet as our choice of
feature extractor. The framework is trained for 80 epochs, and the parameter
aggregation is performed every 2 epochs. In the evaluation phase, our method
achieves generalized results on the less represented sites (sites 1 and 2) and
also keeps a comparable performance on the most represented site (site 3).

3rd Rank (Algo. #3)
Simple yet Effective Federated Learning Workflow for Non-iid Breast Density
Classification with CommunicationLimitation
Yaojun Hu1, Haochao Ying1,2, Yuyang Xu1,3

1Real Doctor AI Research Centre, Zhejiang University, China
2School of Public Health, Zhejiang University, China
3College of Computer Science and Technology, Zhejiang University, China

Accurately estimating breast density is of significant value in early
detection of masked breast cancer. However, in this competition, we observe
two important issues. First, the multi-institutional competition dataset is
not independently and identically distributed. Second, Competition training
and testing environments have communication limitations. To this end, we
employ federated averaging workflow, exactly SCAFFOLD algorithm, for data
heterogeneity issue, along with Top-k gradient sparse for communication cost.
Basically, DenseNet-121 model is utilized as the base model. Finally, we rank
third with the average performance of 0.626 in linear kappa, 0.729 in quad
kappa, and 0.928 in AUC at three testing institutional data, respectively.

4th Rank (Algo. #4)
Breast Density Classification Using Federate Learning Using Pretrained Breast
Density Classifiers
Conrad Testagrose1, Mutlu Demirer2, Vikash Gupta2, Xudong Liu1, Barbaros S.
Erdal2, Richard D. White2

1University of North Florida College of Computing, Jacksonville, United States
2Mayo Clinic Florida Radiology - CAII, Jacksonville, United States

We used transfer learning for performing this classification challenge.
Inception-V3 [Szegedy et al., 2016] is used as the base network with weights
from ImageNet [Alex Krizhevsky, Ilya Sutskever, Geoffrey E Hinton, 2017] as
initialization. The images were resized to a resolution of 299 x 299 and the
single grayscale channel was converted to a three-channel image (similar to
color images in the ImageNet [Alex Krizhevsky, Ilya Sutskever, Geoffrey E
Hinton, 2017] dataset). This model was further fine-tuned using data from the
Mayo Clinic Health system. The Mayo Clinic data used was comprised of 118,459
total mammogram images. Of these images, 55826 were C-View images (Hologic,
Marlborough, MA, https://www.hologic.com/), 49040 were Full Field Digital
Mammography images, and 13,593 were Tomosynthesis Projection images. This
model was then further fine-tuned using the training data provided for the
challenge. Data preprocessing techniques similar to that used on the Mayo
Clinic data were used on the challenge training data. Data augmentation
techniques such as image rotation, flipping, and normalization were performed
on the images in this preprocessing step. The submitted model was trained
using a learning rate of 0.001 with a batch size 16 and the Adam Optimizer
[Kingma and Ba, 2014]. The training pipeline tools were built using PyTorch
[Paszke et al., 2019], NVFlare [Roth et al., 2022], and the MONAI [Cardoso et
al., 2022] machine learning toolkit.

5th Rank (Algo. #6)
FedAvg-ed ResNet18 for Breast Density Classification
Ünal Akünal∗, Markus Bujotzek∗, Klaus H. Maier-Hein

Division of Medical image Computing, German Cancer Research Center,
Heidelberg, Germany
∗Equal contribution

We recognized the similarity between the conducted Federated Learning
challenge for breast density classification and the work by Roth et al.
[2020]. Inspired by it, we started the federated training with a FedAvg
aggregation strategy for 150 federated communication rounds while performing 1
local epoch per round. As a model, DenseNet121 was trained via a CE-loss and
Adam optimizer (lr=1e-4) with a batch size of 64. This SOTA setup failed for
the given challenge due to the boundary condition that a training should not
last longer than 8 hours. Moreover, we found in further experiments that
models larger than ResNet50 couldn’t be handled by the provided training
infrastructure due to the computational constraints on message size. To reduce
both training duration and model size, we selected ResNet18 as a model trained
with a batch size of 64 via CE-loss and SGD optimizer (lr=2e-3, momentum=0.9)
to achieve faster convergence in the limited training time. We stuck to FedAvg
as an aggregation strategy, but reduced the number of federated communication
rounds to 100 while still performing 1 local epoch per communication round.
With this setup we also achieved the best training results measured by the
applied metrics and therefore submitted this setup for testing. We finalized
our investigations by modifying FedAvg to FedProx method to tackle a
potentially present domain shift between the dataset splits. However, since
our FedProx setup performed worse than the previously described setup, we
concluded that the present domain shift between the data splits is not as
heterogenous in the sense that the proximal term in FedProx does not
contribute to but hinders the performance of the global model compared to the
previous FedAvg experiments.

6th Rank (Algo. #5)
Untitled submission
Ghanem Bahrini

Université de Carthage, Tunis, Tunesia

[Note from challenge organizers] The author did not submit a description of
this method. However, we include the result for completeness as it was used in
the final ranking of the challenge. The method is based on the baseline
implementation provided by the challenge organizers with only a slight
modification, i.e., using ResNet50 instead of the baseline ResNet18.

7th Rank (Algo. #7)
Fine-tuned FedAvg for Federated Breast Density Estimation
Yi Qin, Xiaomeng Li

Electronic and Computer Engineering, Hong Kong University of Science and
Technology

Accurate breast density estimation is essential in early breast cancer
diagnosis. Federated learning provides privacy-preserving solutions for
precisely estimating the density from multiple isolated data sources. In this
challenge, we finetuned the FedAvg algorithm to provide a solution for the
federated breast density estimation task. We used the local data and their
partition from the organizer as our client’s input. First, each client trained
their local ResNet-56 model to minimize the cross entropy loss for five
epochs. Then, clients uploaded the parameters of their local encoder to the
server. After this, the server aggregated a new global model by weighted
averaging the received clients’ model weight. The weight is determined by the
proportion of the local client’s data size to the total data amount. Finally,
the server updated the local models by distributing the global model to local
clients. We deployed a grid search to derive the optimal hyperparameters. The
submitted results achieved AUC all 0.899, Lin. Kappa all 0.59, and Quad. Kappa
all 0.698.

##  Appendix B Additional Ranking Analysis

To summarize the statistical ranking of the finalist algorithms, we show their
ranking stability over all per-image metric tasks in Fig. 9. Here, the plots
show results from multiple tasks separated by algorithm, providing more
insight into the assessment data. This can help to better understand the
characteristics of each task and the level of uncertainty in ranking the
algorithms for each task.

Fig. 9: Overall Ranking stability across different metrics and challenge
tasks. Figures are ordered from first to last rank, going from left to right,
top to bottom.