[1]Vishwa S. Parekh

1]University of Maryland Medical Intelligent Imaging (UM2ii) Center,
University of Maryland School of Medicine, Baltimore, MD

# Surgical Aggregation: A Collaborative Learning Framework for Harmonizing
Distributed Medical Imaging Datasets with Diverse Tasks

Pranav Kulkarni<EMAIL_ADDRESS>Adway Kanhere
<EMAIL_ADDRESS>Paul H. Yi<EMAIL_ADDRESS><EMAIL_ADDRESS>[

###### Abstract

Large-scale chest x-ray datasets have been curated for the detection of
abnormalities using deep learning, with the potential to provide substantial
benefits across many clinical applications. However, each dataset focuses only
on a subset of findings that can be simultaneously present in a patient,
making it challenging to train models that aggregate multiple datasets
together. Therefore, data harmonization is crucial to leverage these datasets
in aggregate to train clinically useful models with a complete representation
of abnormalities that may occur within the thorax. To that end, we propose
surgical aggregation, a collaborative learning framework for harmonizing and
aggregating knowledge from distributed heterogeneous datasets with partial
annotations. We evaluate surgical aggregation across synthetic and real-world
heterogeneous datasets with partial annotations. Our results indicate that
surgical aggregation outperforms current strategies, generalizes better, and
has the potential to facilitate the development of clinically useful models
even when using datasets with heterogeneous disease labels.

###### keywords:

Deep learning, federated learning, surgical aggregation, data harmonization,
chest x-ray, classification.

## Main

Figure 1: Siloed approach of developing and training models separately on
large-scale medical imaging datasets. In this illustration, both institutes
curate chest x-ray datasets to train deep learning models on similar tasks.
However, due to data and label heterogeneity and patient data privacy,
harmonizing and leveraging knowledge from both datasets is difficult. (Icons
from Flaticon.com)

Chest X-Ray (CXR) is the most commonly ordered medical imaging study globally
and is critical for screening many life threatening conditions. Many large-
scale curated CXR datasets have been released for the purpose of training
clinically useful deep learning models with expert-level detection of diseases
in the thorax [1, 2, 3, 4]. Unfortunately, due to the nature of curating
large-scale datasets, each dataset focuses only on detecting a subset of
findings that can be simultaneously present in a patient, thereby limiting
their clinical utility. For example, the NIH Chest X-Ray 14 [4] and CheXpert
[1] datasets contain 14 and 13 labels respectively, but share only seven
common disease labels. Since annotations present in one dataset may not be
present in another, these datasets are incomplete due to this inherent label
heterogeneity. Similarly, each dataset is acquired and curated using different
imaging equipment, acquisition parameters, image processing techniques and
label extraction methods – making them heterogeneous and not independent and
identically distributed (iid). To leverage knowledge from these datasets would
require training models separately on each dataset. This siloed approach would
result in an ever increasing number of models as more datasets get curated and
released, thus, making it impractical. Furthermore, due to domain shift,
individually trained models may not generalize well in the real-world to
unseen data sourced from another institution [5, 6, 7, 8, 9, 10].

Different groups have explored the idea of building generalized multi-task
deep learning models for different applications by leveraging multiple
datasets in aggregate. Cohen et al. [11] observed that models trained on
multiple datasets in aggregate demonstrated higher performance and generalized
better to unseen datasets, compared to models trained individually. However,
aggregating datasets in a centralized training setup is not only expensive due
to storage and bandwidth constraints, but also practically challenging due to
patient data privacy concerns. Therefore, federated learning (FL) was proposed
as promising collaborative machine learning paradigm that approaches this
problem from a decentralized and distributed approach. Consisting of a central
server with multiple participating clients, FL can learn a global model on
distributed datasets by aggregating knowledge learned by local models at each
client. In the realm of medical imaging, FL has enabled training of large-
scale global models using homogeneous data spread across multiple institutions
without sharing sensitive patient data [12, 13, 14]. However, majority of the
datasets being curated across the world are non-iid, heterogeneous and focus
on similar but different tasks. Since local models may differ significantly
from each other, data and label heterogeneity poses a significant challenge to
leveraging knowledge from these datasets in aggregate using the federated
approach.

Figure 2: An overview of the surgical aggregation framework. Due to inherent
differences in image acquisition, annotation, and curation, large-scale
medical imaging datasets are heterogeneous and focus on similar but different
disease annotations. Surgical aggregation harmonizes and aggregated knowledge
from these distributed heterogeneous datasets into a global deep learning
model. (Icons from Flaticon.com)

Different strategies and heuristics have been proposed in FL literature for
aggregating knowledge from heterogeneous non-iid datasets and tackle domain
shift [15, 16, 17, 18, 9, 19, 20]. For example, knowledge distillation has
emerged as a popular technique across multiple domains to tackle data
heterogeneity [21, 16, 17, 22, 20]. An ensemble attention distillation based
FL approach was explored to harmonize the NIH and CheXpert datasets,
demonstrating the ability to train a global model with 14 disease labels [17].
However, the proposed approach required an external dataset and a patient
could not be diagnosed with more than one disease, making it practically
challenging and not clinically useful. Meanwhile, other groups have pin-
pointed the source of domain shift in the federated setup to the presence of
batch normalization (BN) layers, a core element within model architectures
[20, 23, 9, 24]. By normalizing layer inputs, BN aims to ensure rapid and
stable model convergence by learning a moving mean of the training data
statistics [25]. However, in a federated setup, the naïve aggregation of these
statistics – in the case of FedAvg [26], the de-facto strategy for model
weight aggregation – has been demonstrated to yield a sub-optimal solution
with distributed non-iid data due to domain shift [20, 23]. Consequently,
various groups have explored newer weight aggregation strategies to address
the impact of data heterogeneity in the federated setup [20, 9, 24].

Data and label harmonization are two sides of the same coin. While significant
advancements in recent literature have aimed to address data heterogeneity, it
is evident that the presence of incomplete annotations in medical imaging data
is an impediment to leveraging these large-scale heterogeneous datasets in
aggregate across both federated and centralized setups. Most approaches focus
on either detecting either (1) the presence of an abnormality i.e. normal vs.
not normal, or (2) the disease labels shared across all datasets [10, 8, 11,
27]. As a result, the clinical utility of training a model across multiple
datasets in aggregate is limited without data and disease label harmonization.
Therefore, a method to harmonize and aggregate knowledge from distributed
datasets with partial annotations could revolutionize the development of
clinically-relevant deep learning models. To that end, we propose surgical
aggregation, a collaborative learning framework for harmonizing and
aggregating knowledge from distributed heterogeneous datasets with partial
disease annotations into a global deep learning model. As visualized in Figure
2, we utilize surgical aggregation in the central server to aggregate relevant
knowledge coming in from each client to effectively learn different tasks
across distributed heterogeneous datasets. Our proposed framework is (1) model
and task-agnostic, (2) semi-supervised i.e. does not require any additional
external dataset for validation during the training process, (3) capable of
training a global model across all observed disease labels, and (4) can work
in a federated and centralized setup.

Figure 3: Comparison between the mean AUROC scores and BCE loss metrics of
surgical aggregation, central aggregation, federated learning, and baseline
models on the held-out NIH test set when evaluating the effect of (a) number
of clients ($K$), (b) label heterogeneity ($\psi$), and (c) dataset size
($N$).

## Results

In this work, we evaluated surgical aggregation across two experiments for the
detection of abnormalities in CXRs. In the first experiment, we utilized
synthetic iid datasets sampled from the NIH dataset to perform analytical
experiments to evaluate the performance of surgical aggregation across varying
data and disease label heterogeneity and dataset size. In the second
experiment, we surgically aggregated and harmonized the NIH and CheXpert
datasets, two real-world non-iid datasets with partial labels, into a global
model with the capability to classify all 20 observed disease labels. Across
both experiments, we compared surgical aggregation with two other strategies
for training models in aggregate with the presence of partial annotations:

  1. 1.

Central Aggregation, where contributing datasets are assumed to be locally
available and the model is trained in a centralized setup across all observed
labels.

  2. 2.

Federated Learning, where contributing datasets are distributed across
multiple clients and the model is trained using the conventional FL approach
where the entire network is aggregated across all participating clients
without surgical aggregation.

### Analytical experiments with synthetic iid datasets

For our first set of experiments, we sampled multiple synthetic iid datasets
from the NIH dataset to simulate different number of clients as well as
different levels of data and label heterogeneity. The goal of these
experiments was to explore the scalability of surgical aggregation across
various real-world setups by varying the following factors:

Figure 4: Comparison between the mean AUROC scores and BCE loss metrics of
surgical aggregation, central aggregation, federated learning, and baseline
models on the external MIMIC test set when evaluating the effect of (a) number
of clients ($K$), (b) label heterogeneity ($\psi$), and (c) dataset size
($N$).

  1. 1.

Number of participating clients ($K$)

  2. 2.

Label heterogeneity ($\psi$)

  3. 3.

Dataset size ($N$)

We used the baseline deep learning model that was trained using the entire NIH
dataset and complete disease label representation to establish an upper bound
for model performance. This allowed us to evaluate all the approaches on the
held-out NIH test set containing all disease labels. Across all three factors,
we hypothesize that as each client’s contributions to the complete data and
disease label representation become sparse and model convergence becomes
significantly harder to achieve. Our results demonstrate that surgical
aggregation consistently outperformed all other approaches on the held-out NIH
test set ($N=22,330$) as well as the external MIMIC test set ($N=377,110$)
across varying number of clients, label heterogeneity, and dataset size, as
illustrated in Figures 3 and 4.

As shown in Figure 3, the performance margin between surgical aggregation and
federated learning significantly increased with increase in the number of
federated clients ($0.06$ for $K=2$ to $0.14$ for $K=10$) as well as increase
in label heterogeneity ($0$ for $\psi=0$ to $0.12$ for $\psi=1$). Similar
results were also observed in the external MIMIC test set as shown in Figure
4. These results demonstrate the efficacy of using surgical aggregation for
aggregating knowledge from distributed datasets, especially in real-world
scenarios with large number of clients and diverse disease labels. Detailed
breakdown of model performance, along with p-values for statistical
significance, is provided in Extended Data Tables 1, 2, 3. Disease-level model
performance is visualized in Extended Data Figure 6 with a detailed breakdown
provided in Supplementary Section 2.

Overall, our results indicate that surgical aggregation provides a significant
improvement in model performance compared to central aggregation and federated
learning. Across varying number of participating clients, surgical aggregation
yields a mean AUROC of $0.81\pm 0.01$ on the held-out NIH test set, compared
to central aggregation ($0.78\pm 0.02$, $p=0.004$) and federated learning
($0.71\pm 0.05$, $p<0.001$), as visualized in Figure 3(a). Similarly, on the
external MIMIC test set, surgical aggregation has better generalizability with
a mean AUROC of $0.73\pm 0.01$ compared to central aggregation ($0.71\pm
0.01$, $p=0.02$) and federated learning ($0.69\pm 0.02$, $p=0.001$), as
visualized in Figure 4(a). Across varying label heterogeneity, surgical
aggregation yields a mean AUROC of $0.80\pm 0.01$ on the held-out NIH test
set, compared to central aggregation ($0.79\pm 0.02$, $p=0.01$) and federated
learning ($0.74\pm 0.01$, $p=0.01$), as visualized in Figure 3(b). Similarly,
on the external MIMIC test set, surgical aggregation has better
generalizability with a mean AUROC of $0.73\pm 0.01$ compared to central
aggregation ($0.72\pm 0.02$, $p=0.12$) and federated learning ($0.69\pm 0.04$,
$p=0.03$), as visualized in Figure 4(b). Across varying dataset sizes,
surgical aggregation yields a mean AUROC of $0.79\pm 0.02$ on the held-out NIH
test set, compared to central aggregation ($0.77\pm 0.02$, $p=0.001$) and
federated learning ($0.74\pm 0.02$, $p=0.001$), as visualized in Figure 3(c).
Similarly, on the external MIMIC test set, surgical aggregation has better
generalizability with a mean AUROC of $0.72\pm 0.02$ compared to central
aggregation ($0.71\pm 0.02$, $p=0.17$) and federated learning ($0.67\pm 0.03$,
$p=0.004$), as visualized in Figure 4(c).

Figure 5: Comparison between the mean AUROC score metrics of surgical
aggregation, central aggregation, federated learning, and baseline models on
held-out NIH and CheXpert test sets and external MIMIC test set across (a) the
seven shared labels and (b) all observed disease labels.

### Harmonization of NIH and CheXpert datasets

While the results from the analytical experiments using synthetic iid datasets
indicate that surgical aggregation outperforms other strategies, it is crucial
to demonstrate its ability to harmonize real-world non-iid distributed
datasets. In this experiment, we surgically aggregate the NIH and CheXpert
datasets to train a 20 disease label classifier. Unlike the initial
experiments, the complete disease label representation is not available and,
as a result, model performance is measured by the performance of each model on
held-out test sets from the NIH ($N=22,330$) and CheXpert ($N=45,208$)
datasets, in addition to the external MIMIC test set ($N=377,110$). However,
we can still establish an upper bound for model performance by training
baseline models separately on the NIH and CheXpert datasets. We evaluate the
performance of each model across (1) the seven shared labels between the NIH
and CheXpert dataset and (2) all the observed 20 labels.

For the seven shared disease labels, visualized in Figure 5(a), the surgical
aggregation model classified the held-out NIH test set (left) with a mean
AUROC of 0.84, significantly better than the NIH baseline model $(\text{mean
AUROC}=0.83$, $p=0.02$) and the CheXpert baseline model ($\text{mean
AUROC}=0.82$, $p<0.001$). Similarly, the surgical aggregation model classified
the CheXpert test set (middle) with a mean AUROC of 0.77, comparable to the
CheXpert baseline model ($\text{mean AUROC}=0.77$, $p=0.98$), but
significantly better than the NIH baseline model ($\text{mean AUROC}=0.68$,
$p<0.001$). Furthermore, surgical aggregation outperformed all other
approaches across both test sets ($p<0.001$) with the exception of federated
learning on the held-out NIH test set ($p=0.07$). On the external MIMIC test
set (right), we observe that the surgical aggregation model generalizes better
than the baseline models and the other approaches with a mean AUROC of 0.81
($p<0.001$). While the difference in model performance across all models may
be minimal for seven shared disease labels, the superiority of surgical
aggregation is clearly demonstrated when all the observed disease labels are
considered in Figure 5(b). On the held-out NIH test set (left), surgical
aggregation performs comparably to the NIH baseline with a mean AUROC of 0.81
($p=0.06$), while outperforming both central aggregation and federated
learning ($p<0.001$). On the held-out CheXpert test set (middle), surgical
aggregation outperforms the CheXpert baseline model and all other approaches
with a mean AUROC of 0.75 ($p<0.001$). Similarly, we observe that surgical
aggregation outperforms all baseline models, central aggregation, and
federated learning on the external MIMIC test set (left) with a mean AUROC of
0.75 ($p<0.001$). Figure 5 illustrates the performance of all the approaches
across both shared-labels and all-label evaluations. As shown in Figure 5, the
performance of the surgical aggregation model across the two evaluations had a
performance margin of 0.03, significantly lower compared to 0.11 for central
aggregation and 0.14 for federated learning. These results indicate the
potential bias in the performance of federated learning and central
aggregation models towards shared labels as opposed to the unshared labels
between the two datasets. Detailed breakdown of model performance, along with
p-values for statistical significance, is provided in Extended Data Table 4.
Disease-level model performance is visualized in Extended Data Figure 7 with
detailed breakdown provided in Supplementary Section 3.

## Discussion

Although the release of large-scale CXR datasets has spearheaded the utility
of deep learning models for expert-level detection of diseases in the thorax,
the heterogeneous nature of these datasets has limited their clinical utility.
While literature has emphasized the need for better techniques to tackle data
heterogeneity and its consequent domain shift, there is a critically unmet
need for tackling disease label heterogeneity in medical imaging datasets. The
current status quo deregards the presence of heterogeneous disease labels by
either (1) ignoring granular disease annotations (i.e., focusing on normal vs
abnormal) or (2) focusing on the common annotations between datasets [10, 8,
11, 27]. Gong et al. [17] deviated from this status quo by proposing an
ensemble attention distillation based FL approach to harmonize the 14 disease
labels across both NIH and CheXpert datasets. However, their approach required
an external dataset for supervision and a patient could not be diagnosed with
more than one disease, making it practically challenging and not clinically
useful. In this work, we demonstrate how surgical aggregation can fill this
void and tackle disease label heterogeneity with a semi-supervised, data
driven federated approach to harmonize and leverage heterogeneous medical
imaging datasets with partial labels in aggregate to train clinically useful
models.

Across all analytical experiments, we observe that surgical aggregation scales
well to different federated setups with varying data and disease label
heterogeneity, while outperforming centrally aggregated and federated learning
models. We observe similar findings when surgically aggregating the NIH and
CheXpert datasets. Surgical aggregation demonstrates a significant leap in
model performance when all disease labels are considered while generalizing
better to unseen external datasets. We believe this is a combination of two
effects: (1) unlike the other two approaches, surgical aggregation is less
susceptible to degrading model performance due to sparse and varying client
contributions, and (2) surgical aggregation is able to learn a generalized
distribution across diverse datasets and tasks. These findings are in stark
contrast to observations regarding models trained using central aggregation
and federated learning. Firstly, both approaches result in degrading model
performance due to the presence of partial labels. While model performance on
the seven shared labels is close, albeit inferior, to the performance of
surgical aggregation, when all the 20 observed disease labels are considered,
the difference is striking. Unlike surgical aggregation, we observe that
central aggregation and federated learning result in a significant degradation
of performance on disease labels not shared across participating clients, thus
resulting in a significantly worse overall model performance. Finally, our
results indicate that central aggregation is prone to resulting in a model
biased towards the data with greater contributions. We observe that the
centrally aggregated model learned an unequal distribution biased towards the
larger CheXpert dataset and thus, resulting in a significantly worse
performance on the NIH dataset.

To that end, our work demonstrates how surgical aggregation can leverage
large-scale, distributed, non-iid medical imaging datasets with diverse tasks
in aggregate to train clinically useful global models through semi-supervised
data harmonization. In other words, surgical aggregation enables each
participating client to train a robust, well generalizable model on client-
specific tasks using knowledge learned by other clients performing similar
tasks in aggregate, while enabling access to a larger set of different
diagnostic tasks via the global model. Let’s consider a consortium of two
institutions, where the first focuses on the segmentation of multiple tumors
within the abdomen (liver, spleen, etc.), while another focuses solely on
liver tumors. Suppose, in the future, the second institution intends to expand
their diagnostic capabilities to other organs. Due to domain shift, this would
require the second institution to retrain a new model in both centralized and
federated setups. In stark contrast, surgical aggregation addresses this
limitation by training a global model across all tasks in aggregate. It
enables an ad hoc expansion of diagnostic capabilities at the second
institution by ”importing” tasks learned by the global model without the need
for retraining. Furthermore, this capability enables a new institution to join
this consortium and simply import all relevant diagnostic tasks from the
global model for their clinical application without the need for training any
model. As AI-assisted disease characterization becomes a mainstay in
radiology, we demonstrate how surgical aggregation provides a framework for
greater multi-institutional collaboration to train large-scale clinically
useful models in aggregate, while reducing the barrier to entry for other
institutions – a step forward towards bridging the gap from bench to bedside.

## Methods

### Surgical Aggregation

We developed surgical aggregation as a semi-supervised, model and task-
agnostic framework for surgically, i.e. selectively, aggregating task-specific
weights learned by all participating clients into the global model at the end
of each federated round. Let’s consider $K$ participating clients with
$L_{1},L_{2},...,L_{K}$ diseases labels. By utilizing surgical aggregation,
the global model can learn and predict the presence of all
$\bigcup_{k=1}^{K}L_{k}$ observed disease labels. Despite contributing to the
global model, each participating client does not need to be aware of other
clients or their contributions. In other words, a client can simply choose to
contribute knowledge for its client-specific tasks from its training data
without interacting with any other client, or may choose to import knowledge
for different diagnostic tasks via the global model.

We initialized our surgical aggregation setup as a multi-task and multi-label
deep neural network composed of a central server and multiple clients. The
model consists of two distinct blocks: (1) a representation block and (2) a
task block. In such a setup, the representation block learns a generalized
global representation of the data using extracted feature semantics while the
task block learns to interpret this representation and predict the presence of
abnormalities. During the training phase, the weights of layers corresponding
to the representation block are always aggregated from each client at the
central server using a predefined ClientUpdate and ServerUpdate strategies and
redistributed back to the clients, thereby learning a generalized
representation of the data distribution block. On the other hand, the weights
from each task block are surgically aggregated to harmonize and aggregate
task-related knowledge from each client into a global task block. If a disease
label is shared across one or more clients, the weight and bias applied to the
input of the neuron corresponding to that label in each client’s task block is
aggregated using a predefined surgical aggregation strategy TaskBlockUpdate.
However, if a disease label is not shared across any other client, the weight
and bias remains unchanged. This enables the selective aggregation of the
weights applied to the input of the task block construct a global task block
with the ability to classify across all observed disease labels. The surgical
aggregation framework is described in detail in Algorithm 1.

Input: $K$ clients indexed by $k$ with local labels $L_{k}$,

Model layers indexed by $l$,

Initialized model parameters $w_{0,k}^{(i)}$,

Training epochs before aggregation $E$,

Total federated rounds $N$,

Weight aggregation strategy ClientUpdate and ServerUpdate,

Surgical aggregation strategy TaskBlockUpdate;

$L_{global}\ \leftarrow\ \bigcup_{k=1}^{K}L_{k}$

for  _$\text{each round}\ n=1,2,...,N$_ do

for  _$\text{each client}\ k\ \text{and each layer}\ l$_ do

$w_{n,k}^{(l)}\leftarrow\texttt{ClientUpdate}(l,\ w_{n-1,k}^{(l)})$

end for

if  _$\text{mod}(n,E)=0$_ then

for  _$\text{each client}\ k\ \text{and each layer}\ l$_ do

if  _$\text{layer}\ l\ \text{is not TaskBlock}$_ then

$w_{n,k}^{(l)}\leftarrow\texttt{ServerUpdate}(K,\ l,\ w_{n}^{(l)})$

else

for  _$i\leftarrow 1$ to $\text{length of }L_{global}$_ do

$w_{agg}\leftarrow[\ ]$

for  _$\text{each client}\ k$_ do

if  _$L_{global}[i]$ is in $L_{k}$_ then

append $w_{n}^{(l)}$ to $w_{agg}$

end if

end for

$w_{n,k}^{(l)}[:,i]\leftarrow\texttt{TaskBlockUpdate}(w_{agg})$

end for

end if

end for

end if

end for

Algorithm 1 Surgical Aggregation

### Datasets

#### NIH Chest X-Ray 14

The NIH dataset [4] consists of 14 disease labels with $N=112,120$ frontal-
view CXRs from $M=30,805$ patients. We randomly divided the dataset into
training (70%, $N=78,571$, $M=21,563$), validation (10%, $N=11,219$,
$M=3,080$), and testing (20%, $N=22,330$, $M=6,162$) splits while ensuring no
patient appears in more than one split. Furthermore, we sampled two synthetic
iid toy datasets, NIH (1) and (2), from the NIH dataset by equally
distributing patients diagnosed with at least one positive finding with
healthy patients to prevent patient overlap. Both synthetic datasets consisted
of the same number of patients with distinct training ($M=10,781$) and
validation ($M=1,540$) sets sampled from the global splits, while ensuring
there is no patient overlap.

#### CheXpert

The Stanford CheXpert dataset [1] consists of 13 disease labels, where seven
are shared with the NIH dataset, with $N$=224,316 CXRs from $M$=65,240
patients. Since our study focuses on a multi-label setup, we dropped the label
for normal in our analyses. To handle uncertain labels in the dataset, we
chose the U-Zeros approach i.e. treating all uncertain labels as negatives. We
randomly divided the dataset into training (70%, $N=155,470$, $M=45,178$),
validation (10%, $N=22,736$, $M=6,454$), and testing (20%, $N=45,208$,
$M=12,908$) splits while ensuring no patient appears in more than one split.

#### MIMIC-CXR-JPG

To evaluate the generalizability of our methods, we utilized the entire MIMIC-
CXR-JPG dataset [28, 2] as our external test set. Similar to the CheXpert
dataset, the MIMIC dataset consists of 13 disease labels, where seven are
shared with the NIH dataset, with $N=377,110$ CXRs from 65,379 patients. Since
our study focuses on a multi-label setup, we dropped the label for normal in
our analyses. To handle uncertain labels in the dataset, we chose the U-Zeros
approach i.e. treating all uncertain labels as negatives.

### Deep Learning Setups

In our study, we utilized the DenseNet121 model architecture [29] to train a
multi-label classifier using TensorFlow 2.8.1. All models were trained on CXRs
downsampled to $224\times 224$ resolution and normalized between 0 and 1 with
data augmentations and learning rate of 5e-5.

#### Surgical Aggregation

We implemented the surgical aggregation framework with the final fully-
connected Dense layer acting as the task block. We utilized FedFBN [9] to
define weight aggregation strategies ClientUpdate and ServerUpdate. As a
result, we initialized our model architecture using pre-trained ImageNet [30]
weights, warmed-up classification heads for all participating clients prior to
federation using transfer learning, and kept all batch normalization layers
frozen during federation. Finally, we performed surgical aggregation on the
final classification Dense layer, where TaskBlockUpdate would aggregate and
average the weights for each disease label. In other words, if more than one
client contributed knowledge for a disease label, the weights would be
averaged across the contributions of each client.

To compare the performance of surgical aggregation, we trained the following
multi-label classifiers using data contributed by all participating clients in
aggregate with the presence of partial annotations:

#### Central Aggregation

The central aggregations strategy involves the training of a model in a
centralized setup, where all the contributing data is assumed to be locally
available. Despite the previously discussed limitations of this method,
centrally training a model on multiple datasets in aggregate is a common
practice to train robust and well generalizable models [10, 8, 11, 27]

#### Federated Learning

We use a modified federated learning strategy to train a global model across
distributed data with partial annotations. We assume that all contributing
data is distributed identically to the surgical aggregation setup and we
utilize FedFBN [9] for model weight aggregation. However, no surgical
aggregation is performed. To tackle partial annotations, we utilize a modified
binary cross-entropy (BCE) loss function to calculate the partial loss at each
client using only the client-specific disease labels. While each local model
will optimize model performance on observed disease labels, this approach has
an additional overhead of requiring all participating clients to know about
other clients and their contributions.

For all experiments, we utilized the DenseNet121 model architecture [29] to
train a multi-label classifier using TensorFlow 2.8.1. All models were trained
on CXRs downsampled to $224\times 224$ resolution and normalized between 0 and
1 with data augmentations and learning rate of 5e-5. The surgical aggregation
and federated learning models were trained with a batch size of 64 using an
initial learning rate of 5e-5 for 150 epochs with 1 epoch between two
consecutive communication rounds ($E=1$) and early stopping after 25 epochs
with no improvement. At the end of each communication round, local losses from
each participating clients were communicated back to the central server and
the best performing global model was determined by the model with smallest
mean loss. Similarly, the centrally aggregated models were trained with the
same hyperparameters in a centralized setup, assuming all the data is locally
available.

### Model Metrics and Statistical Significance

Across all experiments, we measured model performance using binary cross-
entropy (BCE) loss and the area under the receiver operating characteristic
curve (AUROC). In a multi-label scenario, the AUROC score is defined as the
mean AUROC score across all disease labels. Furthermore, we calculated the 95%
confidence interval (CI) for the AUROC scores and compared one-vs-all using
bootstrapping and a paired two-tailed t-test. We defined statistical
significance for all experiments as $p<0.05$.

### Experimental setup

In this work, we evaluated surgical aggregation for the detection of
abnormalities in CXRs using distributed heterogeneous data with partial
annotations across the following two experiments:

#### Analytical experiments with synthetic iid datasets

In the first experiment, we sampled synthetic iid datasets from the large-
scale NIH dataset by equally distributing patients diagnosed with at least one
positive finding with healthy patients to prevent patient overlap. The size of
each synthetic dataset was determined by the number of participating clients
($K$) while ensuring all synthetic datasets considered of the same number of
patients with distinct training and validation sets without any patient
overlap. More specifically, for a $K$-client setup, the NIH training and
validation sets were divided into $K$ groups with random sampling with each
training set consisting of $M=\lfloor\frac{21,563}{K}\rfloor$ patients and
each validation set consisting of $M=\lfloor\frac{3,080}{K}\rfloor$ patients.
Furthermore, to simulate label heterogeneity, we randomly pruned disease
labels from each synthetic datasets while ensuring the global model learns
knowledge for all 14 disease labels in the NIH dataset.

The motivation behind this setup was to provide the complete disease label
representation via the NIH dataset while each synthetic dataset provided a
glimpse into it. This enabled us to establish an upper bound for model
performance by training a baseline model on the complete NIH dataset and
allowed us to evaluate on the held-out NIH test set containing all disease
labels, in addition to seven labels shared with the external MIMIC test set.
While the goal of this experiment is not to compare with the baseline model,
it still provides valuable insight into the impact of data and label
heterogeneity. More importantly, using these synthetic iid datasets, we
simulated scenarios with distributed data with partial labels to explore the
scalability of surgical aggregation across various setups in the real-world by
varying the following factors:

  1. 1.

Number of participating clients ($K$): In our experimentation, as the number
of participating clients increase, the data becomes increasingly heterogeneous
and individual client contributions become sparse. We compared the performance
of surgical aggregation with resultant models from central aggregation and
federated learning across seven runs with $K=2,3,4,5,6,8,10$.

  2. 2.

Label heterogeneity ($\psi$): To evaluate the effect of label heterogeneity on
surgical aggregation, we compared the performance of the surgically aggregated
model with the other models across increasing label heterogeneity. In this
work, we define label heterogeneity ($\psi$) as the proportion of labels not
shared by the clients. For example, $\psi=0$ is equivalent to every client
contributing knowledge for the same disease labels, while $\psi=1$ refers to
no disease labels being shared by any client. Furthermore, $\psi=0$ is the
best-case scenario. Consequently, all three methods are equivalent and we can
expect similar performance. We evaluated the effect of label heterogeneity in
a four-client ($K=4$) setup across seven runs with
$\psi=0,\frac{2}{14},\frac{6}{14},\frac{10}{14},\frac{12}{14},\frac{13}{14},1$.

  3. 3.

Dataset size ($N$): While the first two factors evaluated the effect of data
and label heterogeneity on surgical aggregation, we evaluate the effect of
dataset sizes with constant data and label heterogeneity. This will enable us
to pin-point whether the source of possible model degradation was due to
heterogeneity or simply a smaller training set. With a constant two-client
($K=2$) setup and label heterogeneity $\psi=\frac{4}{14}$, we evaluate the
impact of varying dataset size across six runs with
$N=\frac{1}{10},\frac{1}{8},\frac{1}{6},\frac{1}{4},\frac{1}{3},\frac{1}{2}$.

Disease label distributions for this experiment are provided in Supplementary
Section 1.

#### Harmonization of NIH and CheXpert datasets

While the analytical experiments using synthetic iid datasets provide
invaluable insight into the scalability of surgical aggregation across varying
setups, they do not reflect a real-world scenario with large-scale non-iid
datasets. Out of all the possible scenarios accounted for our study, tackling
distributed non-iid data with partial annotations is the most complex problem
due to significant domain shift. In this experiment, we utilize surgical
aggregation to harmonize and aggregate the NIH and CheXpert datasets to train
a 20 disease label classifier. This is equivalent to a two-client ($K=2$)
setup with label heterogeneity $\psi=\frac{7}{20}$ ($\approx 35\%$). During
training, we utilize differential learning rates $1e-5$ and $5e-5$ for the NIH
and CheXpert datasets respectively. However, unlike the analytical
experiments, the complete representation of all 20 disease labels is not
available. Consequently, we measure model performance individually on the
held-out NIH and CheXpert test sets, in addition to the external MIMIC
dataset. Additionally, we cross-evaluate the NIH baseline model on the
CheXpert test set and vice versa. With these evaluations, we measure two
distinct result: the performance of the models on (1) the seven disease labels
shared by the NIH and CheXpert datasets and (2) all the 20 observed disease
labels. This was motivated to analyse the efficacy of surgical aggregation on
the knowledge shared among datasets as well as the knowledge not shared.

## Supplementary information

This manuscript is accompanied by a single supplementary materials file
containing supplementary sections 1-3 and tables 1-22.

## Declarations

Some journals require declarations to be submitted in a standardised format.
Please check the Instructions for Authors of the journal to which you are
submitting to see if you need to complete this section. If yes, your
manuscript must contain the following sections under the heading
‘Declarations’:

  * •

Funding: N/A

  * •

Competing interests: No competing interests

  * •

Ethics approval: N/A

  * •

Consent to participate: N/A

  * •

Consent for publication: N/A

  * •

Data availability: In our study, three publicly available datasets were used
for analysis.

    * –

NIH Chest X-Ray 14: No restrictions, in public domain,
https://arxiv.org/abs/1705.02315, https://nihcc.app.box.com/v/ChestXray-
NIHCC/folder/36938765345

    * –

CheXpert: Restricted under Stanford University Dataset Research Use Agreement
due to patient health data privacy,
https://stanfordmlgroup.github.io/competitions/chexpert

    * –

MIMIC-CXR-JPG: Restricted under PhysioNet Credentialed Health Data License
1.5.0 due to patient health data privacy, https://physionet.org/content/mimic-
cxr-jpg/2.0.0

  * •

Code availability: Code used in this study is available under the GNU General
Public License v3.0 on https://github.com/UM2ii/SurgicalAggregation

## References

  *   * Irvin et al. [2019] Irvin, J., Rajpurkar, P., Ko, M., Yu, Y., Ciurea-Ilcus, S., Chute, C., Marklund, H., Haghgoo, B., Ball, R., Shpanskaya, K., et al.: Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, pp. 590–597 (2019)
  * Johnson et al. [2019] Johnson, A.E., Pollard, T.J., Greenbaum, N.R., Lungren, M.P., Deng, C.-y., Peng, Y., Lu, Z., Mark, R.G., Berkowitz, S.J., Horng, S.: Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042 (2019)
  * Nguyen et al. [2022] Nguyen, H.Q., Lam, K., Le, L.T., Pham, H.H., Tran, D.Q., Nguyen, D.B., Le, D.D., Pham, C.M., Tong, H.T., Dinh, D.H., et al.: Vindr-cxr: An open dataset of chest x-rays with radiologist’s annotations. Scientific Data 9(1), 1–7 (2022)
  * Wang et al. [2017] Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M.: Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2097–2106 (2017)
  * Guan and Liu [2021] Guan, H., Liu, M.: Domain adaptation for medical image analysis: a survey. IEEE Transactions on Biomedical Engineering 69(3), 1173–1185 (2021)
  * Castro et al. [2020] Castro, D.C., Walker, I., Glocker, B.: Causality matters in medical imaging. Nature Communications 11(1), 3673 (2020)
  * Kilim et al. [2022] Kilim, O., Olar, A., Joó, T., Palicz, T., Pollner, P., Csabai, I.: Physical imaging parameter variation drives domain shift. Scientific Reports 12(1), 21302 (2022)
  * Pooch et al. [2020] Pooch, E.H., Ballester, P., Barros, R.C.: Can we trust deep learning based diagnosis? the impact of domain shift in chest radiograph classification. In: Thoracic Image Analysis: Second International Workshop, TIA 2020, Held in Conjunction with MICCAI 2020, Lima, Peru, October 8, 2020, Proceedings 2, pp. 74–83 (2020). Springer
  * Kulkarni et al. [2023] Kulkarni, P., Kanhere, A., Yi, P.H., Parekh, V.S.: Optimizing federated learning for medical image classification on distributed non-iid datasets with partial labels. arXiv preprint arXiv:2303.06180 (2023)
  * Yao et al. [2019] Yao, L., Prosky, J., Covington, B., Lyman, K.: A strong baseline for domain adaptation and generalization in medical imaging. arXiv preprint arXiv:1904.01638 (2019)
  * Cohen et al. [2020] Cohen, J.P., Hashir, M., Brooks, R., Bertrand, H.: On the limits of cross-domain generalization in automated x-ray prediction. In: Medical Imaging with Deep Learning, pp. 136–155 (2020). PMLR
  * Chowdhury et al. [2022] Chowdhury, A., Kassem, H., Padoy, N., Umeton, R., Karargyris, A.: A review of medical federated learning: Applications in oncology and cancer research. In: International MICCAI Brainlesion Workshop, pp. 3–24 (2022). Springer
  * Rieke et al. [2020] Rieke, N., Hancox, J., Li, W., Milletari, F., Roth, H.R., Albarqouni, S., Bakas, S., Galtier, M.N., Landman, B.A., Maier-Hein, K., et al.: The future of digital health with federated learning. NPJ digital medicine 3(1), 1–7 (2020)
  * Sheller et al. [2020] Sheller, M.J., et al.: Federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. Scientific Reports 10(12598) (2020) https://doi.org/10.1038/s41598-020-69250-1
  * Arasteh et al. [2022] Arasteh, S.T., Isfort, P., Saehn, M., Mueller-Franzes, G., Khader, F., Kather, J.N., Kuhl, C., Nebelung, S., Truhn, D.: Collaborative training of medical artificial intelligence models with non-uniform labels. arXiv preprint arXiv:2211.13606 (2022)
  * Gong et al. [2021] Gong, X., Sharma, A., Karanam, S., Wu, Z., Chen, T., Doermann, D., Innanje, A.: Ensemble attention distillation for privacy-preserving federated learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15076–15086 (2021)
  * Gong et al. [2022] Gong, X., Song, L., Vedula, R., Sharma, A., Zheng, M., Planche, B., Innanje, A., Chen, T., Yuan, J., Doermann, D., et al.: Federated learning with privacy-preserving ensemble attention distillation. IEEE Transactions on Medical Imaging (2022)
  * Kulkarni et al. [2022] Kulkarni, P., Kanhere, A., Yi, P.H., Parekh, V.S.: From competition to collaboration: Making toy datasets on kaggle clinically useful for chest x-ray diagnosis using federated learning. arXiv preprint arXiv:2211.06212 (2022)
  * Kulkarni et al. [2023] Kulkarni, P., Kanhere, A., Yi, P.H., Parekh, V.S.: Surgical aggregation: A federated learning framework for harmonizing distributed datasets with diverse tasks. arXiv preprint arXiv:2301.06683 (2023)
  * Li et al. [2021] Li, X., Jiang, M., Zhang, X., Kamp, M., Dou, Q.: Fedbn: Federated learning on non-iid features via local batch normalization. arXiv preprint arXiv:2102.07623 (2021)
  * Chang et al. [2019] Chang, H., Shejwalkar, V., Shokri, R., Houmansadr, A.: Cronus: Robust and heterogeneous collaborative learning with black-box knowledge transfer. arXiv preprint arXiv:1912.11279 (2019)
  * Li and Wang [2019] Li, D., Wang, J.: Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581 (2019)
  * Qu et al. [2021] Qu, L., Balachandar, N., Rubin, D.L.: An experimental study of data heterogeneity in federated learning methods for medical imaging. arXiv preprint arXiv:2107.08371 (2021)
  * Wang et al. [2023] Wang, Y., Shi, Q., Chang, T.-H.: Why batch normalization damage federated learning on non-iid data? arXiv preprint arXiv:2301.02982 (2023)
  * Ioffe and Szegedy [2015] Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International Conference on Machine Learning, pp. 448–456 (2015). pmlr
  * McMahan and more [2017] McMahan, B., more: Communication-Efficient Learning of Deep Networks from Decentralized Data. In: Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. Proceedings of Machine Learning Research, vol. 54, pp. 1273–1282. PMLR, ??? (2017)
  * Seyyed-Kalantari et al. [2021] Seyyed-Kalantari, L., Zhang, H., McDermott, M.B., Chen, I.Y., Ghassemi, M.: Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations. Nature medicine 27(12), 2176–2182 (2021)
  * Goldberger et al. [2000] Goldberger, A.L., Amaral, L.A., Glass, L., Hausdorff, J.M., Ivanov, P.C., Mark, R.G., Mietus, J.E., Moody, G.B., Peng, C.-K., Stanley, H.E.: Physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. circulation 101(23), 215–220 (2000)
  * Huang et al. [2017] Huang, G., et al.: Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition (2017) https://doi.org/10.48550/arXiv.1608.06993
  * Deng et al. [2009] Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248–255 (2009). Ieee

## Extended Data

Figure 6: Disease-level AUROC scores on the held-out NIH test set when
evaluating the effect of (a) number of clients ($K$), (b) label heterogeneity
($\psi$), and (c) dataset size ($N$).

Figure 7: (a) Disease-level AUROC scores for shared disease labels on the
held-out NIH and CheXpert test sets. (b) Disease-level AUROC scores for NIH
disease labels not shared with the CheXpert dataset on the held-out NIH test
set. (c) Disease-level AUROC scores for CheXpert disease labels not shared
with the NIH dataset on the held-out CheXpert test set.

Table 1: Model metrics for analyzing the effect of number of clients ($K$) on
the held-out NIH test set and external MIMIC test set. 95% CI for mean AUROC
scores provided in brackets with p-values for statistical significance. Best
performing models across each experiment are highlighted in bold.

Table 2: Model metrics for analyzing the effect of label heterogeneity
($\psi$) on the held-out NIH test set and external MIMIC test set. 95% CI for
mean AUROC scores provided in brackets with p-values for statistical
significance. Best performing models across each experiment are highlighted in
bold.

Table 3: Model metrics for analyzing the effect of dataset size ($N$) on the
held-out NIH test set and external MIMIC test set. 95% CI for mean AUROC
scores provided in brackets with p-values for statistical significance. Best
performing models across each experiment are highlighted in bold.

Table 4: Model metrics on held-out NIH and CheXpert test sets and external
MIMIC test set. 95% CI for mean AUROC scores provided in brackets with
p-values for statistical significance. Best performing models are highlighted
in bold.

(a) Seven Shared Disease Labels

(b) All Disease Labels


phase, it simulates $H_{\rho}$ w/o producing anything but by summing vectors
of the form $(\ell,0)$ where $\ell$ is the length of output produced by the
current simulated transition of $H_{\rho}$ (wlog we assume that $H_{\rho}$
outputs at most one symbol at a time, i.e. $\ell\in\\{0,1\\}$). Eventually,
when $H_{\rho}$ triggers a transition producing some $\sigma\in\Gamma$,
$\mathcal{P}$ non-deterministically decides to increment its first component
by $1$, one last time, and stores the symbol $\sigma$ in its state. Then it
proceeds to phase 2, which does exactly the same but on the second vector
components (vectors of the form $(0,\ell)$) and by simulating
$H^{\prime}_{\rho}$ instead of $H_{\rho}$. This is continued till a non-
deterministic choice is made of stopping the increment of the second
component, and storing in its state, the last symbol $\beta$ output by
$H^{\prime}_{\rho}$.

Finally, its set of accepting states are pairs $(\sigma,\beta)$ such that
$\sigma\neq\beta$ and the semi-linear accepting set is defined by the equation
$x=y$. Clearly, $\mathcal{P}$ is finite-visit because $H_{\rho}$ and
$H^{\prime}_{\rho}$, being deterministic two-way transducers, are finite-visit
as well, concluding the proof.


11institutetext: Christian Albrechts University at Kiel, Germany, 11email:
<EMAIL_ADDRESS>
# Disparity among low first ionization potential elements

Verena Heidrich-Meisner 11 Lars Berger 11 and Robert F. Wimmer-Schweingruber
11

###### Abstract

Context. The elemental composition of the solar wind differs from the solar
photospheric composition. Elements with low first ionization potential (FIP)
appear enhanced compared to O in the solar wind relative to the respective
photospheric abundances. This so-called FIP effect is different in the slow
solar wind and the coronal hole wind. However, under the same plasma
conditions, for elements with similar FIPs such as Mg, Si, and Fe, comparable
enhancements are expected.

Aims. We scrutinize the assumption that the FIP effect is always similar for
different low FIP elements, namely Mg, Si, and Fe.

Methods. Here we investigate the dependency of the FIP effect of low FIP
elements on the O7+/O6+ charge state ratio depending on time, that is the
solar activity cycle, and solar wind type. In addition, we order the observed
FIP ratios with respect to the O7+/O6+ charge state ratio into bins and
analyze separately the respective distributions of the FIP ratio of Mg, Si,
and Fe for each O7+/O6+ charge state ratio bin.

Results. We observe that the FIP effect shows the same qualitative yearly
behavior for Mg and Si, while Fe shows significant differences during the
solar activity maximum and its declining phase. In each year, the FIP effect
for Mg and Si always increases with increasing O7+/O6+ charge state ratio, but
for high O7+/O6+ charge state ratios the FIP effect for Fe shows a
qualitatively different behavior. During the years 2001–2006, instead of
increasing with the O7+/O6+ charge state ratio, the Fe FIP ratio exhibits a
broad peak or plateau. In addition, the FIP distribution per O7+/O6+ charge
state bin is significantly broader for Fe than for Mg and Si.

Conclusions. These observations support the conclusion that the elemental
fractionation is only partly determined by FIP. In particular, the qualitative
difference behavior with increasing O7+/O6+ charge state ratio between Fe on
the one hand and Mg and Si on the other hand is not yet well explained by
models of fractionation.

###### Key Words.:

solar wind, elemental composition

##  1 Introduction

The solar elemental composition has been under investigation for several
decades (Feldman, 1992; Schmelz et al., 2012; Meyer, 1985). The photospheric
elemental composition is typically assumed to be constant with the solar
activity cycle and is used as the reference for the elemental composition
observed in active regions (Baker et al., 2015), in coronal loops (Del Zanna &
Mason, 2003), and in the solar wind (Feldman, 1992; Schmelz et al., 2012;
Meyer, 1985). It is well known that some elements are depleted or enriched in
the solar wind compared to the photosphere. Since elements with similar first
ionization potentials (FIPs) show a similar elemental composition in the solar
wind relative to the photosphere, this is called the “FIP effect” and the
underlying mechanism behind this effect separates ions from neutrals, and is
thus assumed to act preferentially on ions rather than on neutral atoms. It is
still unknown whether low FIP elements are enhanced in the solar wind compared
to the photosphere or whether instead high FIP elements are depleted. Both
interpretations are consistent with the observations.

Several theoretical approaches have been explored in modeling the FIP effect,
including quasi-thermal processes of inefficient Coulomb drag/gravitational
settling/diffusion (von Steiger & Geiss, 1989; Marsch et al., 1995; Bochsler
et al., 2000; Pucci et al., 2010; Peter, 1996, 1998), heating by coronal ion
cyclotron waves (Schwadron, 1999), chromospheric reconnection (Arge & Mullan,
1998), electric currents (Feldman & Widing, 2003), and most recently and most
promisingly, the ponderomotive force arising as Alfvén waves propagate through
or reflect from the chromosphere (Doschek et al., 2015; Laming, 2012, 2015;
Dahlburg et al., 2016; Laming, 2017).

The FIP effect is more pronounced in the slow solar wind than in the fast
coronal hole wind (Feldman, 1992; Schmelz et al., 2012; Meyer, 1985). Since
they have similar FIPs, Mg (with a FIP of $7.646$ eV), Si (with a FIP of
$8.151$ eV), and Fe (with a FIP of $7.870$ eV; values taken from Benenson et
al. 2006) are typically assumed to express the same FIP effect in the slow
solar wind and in the coronal hole wind. This is consistent with long-term
averages of the respective elemental abundances (Schmelz et al., 2012) and the
dominant models of the FIP effect (Laming, 2015). However, as discussed in
detail in Laming et al. (2017), the FIP effect is not the only cause of
fractionation in the solar wind. Pilleri et al. (2015) have already observed
considerable fractionation between low FIP elements.

As visible in Figure 1 of Schmelz et al. (2012) and discussed in detail in
Reames (2018), another interesting fractionation case is S (with a FIP of
10.36 eV), which fractionates as a high FIP element in the closed loop solar
corona, but is observed with a higher FIP ratio in the solar wind.

One of the main science goals of the upcoming Solar Orbiter mission (Müller et
al., 2013) is to identify the source regions of the slow solar wind. In tune
with the general scheme of Solar Orbiter to heavily exploit combined
observations of its instruments, a coordinated observation of the elemental
abundance by the SPectral Imaging of the Coronal Environment (SPICE) and the
Heavy Ion Sensor (HIS) is intended as the main tool for investigating
potential slow solar wind source regions. Both instruments can determine the
strength of the FIP effect for several elements, and coordinated observations
of the same plasma packages are planned. However, this relies on the
assumption that in particular the low FIP elements Mg, Si, and Fe show highly
correlated behavior even on short timescales. Here, we utilize the available
observations of the Solar Wind Ion Composition Spectrometer (SWICS) on the
Advanced Composition Explorer (ACE) to scrutinize this assumption and
investigate the differences in the behavior of these low FIP elements.

Figure 1:  FIP ratios per year from 2001 to 2010. The columns show the FIP
ratios for Fe, Mg, and Si. Each row corresponds to the same year. In each
panel, a 2D histogram of all 12-minute observations of FIP ratios vs the
$\log\left(\frac{n^{O7+}}{n^{O6+}}\right)$ from the respective year are shown.
Each vertical slice is normalized to its maximum. The number of contained data
points is given in the inset of each panel. The dashed horizontal line
indicates a FIP ratio of one. The gray solid line marks the median of the FIP
ratio in each respective O charge state ratio bin. Figure 2:  Median FIP
ratios per year from 2001 to 2010 for Fe, Mg, and Si. The median is given only
if at least 100 samples were observed in the respective bin. The colored
shading indicates the 15.9th and 81.1th percentile for each year. In the lower
panels the number of samples per O charge state bin and year are shown for Fe,
Mg, and Si. Figure 3:  Relative densities for pairs of low FIP elements. The
columns correspond to the density ratios $Mg/Fe$, $Si/Fe$, and $Mg/Si$. As in
Figure 1, each panel shows a 2D histogram of 12-minute observations for one
year normalized for each O charge state ratio bin. The insets again specify
the number of data points contained in each panel. Gray shading indicates bins
with significantly different mean densities and the cross-hatching marks bins
where the standard deviation of the two distributions are significantly
different. The dashed horizontal line marks an abundance ratio of one. Figure
4: FIP ratios per solar wind type in a similar format to that in Figure 1
accumulated for the years 2001–2006. Each row shows a different solar wind
type (based on Xu & Borovsky 2015), from top to bottom: all non-ICME solar
wind, coronal hole wind, all slow solar wind, sector reversal plasma, and
helmet streamer plasma. Figure 5: FIP ratios per solar wind type in the same
format as in Figure 4 accumulated for the years 2007–2010.

##  2 Data selection

We are interested in the elemental composition of low FIP elements in the
solar wind and apply our analysis to ten years of data from the Solar Wind Ion
Composition spectrometer (SWICS, Gloeckler et al. 1998) on the Advanced
Composition Explorer (ACE) . The solar wind proton plasma parameters are taken
from the Solar Wind Electron, Proton, and Alpha Monitor (ACE/SWEPAM) (McComas
et al., 1998) and magnetic field observations from the magnetometer ACE/MAG
(Smith et al., 1998).

The ionic composition is derived from the SWICS Pulse Height Analysis (PHA)
words as described in Berger (2008). We use the native 12-minute time
resolution of ACE/SWICS which results in at most 43800 data points in non-leap
years and 43920 in leap years.

The elemental abundance of O, Mg, Si, and Fe is taken as the sum of the
respective most prominent charge states, namely O6+, O7+, Mg7+, Mg8+, Mg9+,
Mg10+, Mg11+, Mg12+, Si7+, Si8+, Si9+, Si10+, Si11+, Si12+, Fe7+, Fe8+, Fe9+,
Fe10+, Fe11+, Fe12+, Fe13+, Fe14+, Fe15+, and Fe16+. Furthermore, to reduce
statistical noise we require for each element a minimum of ten counts
distributed over all respective ions. All data points that violate this
condition are disregarded. However, since this induces a bias against very
thin solar wind conditions, we also verified that the qualitative effect
discussed in this paper remains if data points with fewer counts per element
are included as well. We refer to the ratio of the abundance of an element $X$
with density $n_{X}$ relative to O in the solar wind
$\left(\left(\frac{n_{X}}{n_{O}}\right)_{sw}\right)$ divided by the respective
photospheric ratio $\left(\left(\frac{n_{X}}{n_{O}}\right)_{photo}\right)$
taken from Grevesse & Sauval (1998) as the $X$ FIP ratio
$f(X)=\frac{\left(\frac{n_{X}}{n_{O}}\right)_{sw}}{\left(\frac{n_{X}}{n_{O}}\right)_{photo}}$.

To characterize the solar wind type, we employ the four-type solar wind
categorization scheme from Xu & Borovsky (2015). This heuristic scheme
distinguishes between coronal hole wind, two types of slow solar wind, sector-
reversal plasma and helmet streamer plasma, and ejecta plasma. This solar wind
categorization scheme does not rely on the charge state composition, but only
on (proton) plasma properties of the solar wind. To exclude interplanetary
coronal mass ejections (ICMEs) we use two ICME lists, the Jian et al. (2006,
2011) list and the Richardson & Cane (2010) list, instead of the ejecta
category. We exclude all time intervals from the data set that are considered
as belonging to an ICME in either of the lists. Nevertheless, unidentified
ICMEs are probably still included in the data set. We refer to the union of
the two slow solar wind types as “all slow” solar wind. We repeated our
analysis based on the composition based solar wind categorization from Zhao &
Fisk (2010), and observed the same qualitative behavior.

##  3 Fractionation of low FIP elements

For Ne, a dependence of the elemental abundance on the solar cycle has been
observed (von Steiger & Zurbuchen, 2015). Under the assumption that Ne does
not behave in a fundamentally different way from other elements in the Sun and
the solar wind, this motivates us to independently analyze the FIP effect for
Mg, Si, and Fe for each year.

Figure 1 provides yearly overviews of the observed FIP ratios of Mg, Si, and
Fe from 2001 to 2010 to the logarithm of the respectively observed
O${}^{7}+$/O6+ charge state ratio (we refer to this as the O charge state
ratio only in the following). In the context of the Genesis mission, Pilleri
et al. (2015) carried out a similar study and observed a decrease in the Fe
and Mg FIP ratio with increasing solar wind speeds. Since the O charge state
ratio is likely to be better suited to sort by solar wind type than the solar
wind speed (Zhao & Fisk, 2010), we instead use the O charge state ratio as
reference.

Although all three elements show in all years a wide spread in the observed
FIP ratios, the most frequently occurring FIP ratios range between $1$ and
$4$. For Mg and Si, each year shows a similar pattern: with increasing O
charge state ratio, the FIP ratio increases as well. This is consistent with
the observations that the slow solar wind shows a stronger FIP effect than the
coronal hole wind (as summarized in Schmelz et al. 2012 and Pilleri et al.
2015 with 2–4 for low FIP elements in slow solar wind and 1-2 for coronal hole
wind). For Fe, however, the trend is different, in particular, during the
solar activity maximum (for example in 2004) than during the solar activity
minimum (for example in 2009). In 2009, the Fe FIP ratio shows no trend (or
only a weak one) with increasing O charge state ratio. In 2004, the Fe FIP
ratio exhibits a broad peak around
$\log\left(\frac{n_{O^{7+}}}{n_{O^{7+}}}\right)\sim-0.7$. Also, the Fe FIP
ratio distribution in most O charge state bins appears broader than for Mg and
Si. When using the solar wind speed as a reference value, as in Pilleri et al.
(2015), both the Mg and Fe FIP ratio continue to decrease with increasing
solar wind speed.

Figure 1 further illustrates that the O charge state ratio changes over the
solar cycle. During the solar activity maximum, higher values of the O charge
state ratio are reached, while during solar minimum, lower O charge state
ratio values are observed. Thus, the part of the regime of the O charge state
range which exhibits a broad peak in the Fe FIP ratio from 2001 to 2006, is
only infrequently observed in the following years. Thus, from this figure it
cannot be excluded that the Fe FIP effect always shows the same qualitative
behavior in all years, but this is not visible in 2007–2010 because of
insufficient statistics for this very slow solar wind. Although most ICMEs are
filtered out from the data set, any ICME list can be expected to be
incomplete. Since ICMEs are typically also associated with high charge states
and are more frequent during the solar activity maximum, undetected ICMEs
probably contribute to the broad peak in the Fe FIP ratio. Also, as discussed
in, for example, Klecker et al. (2009), the density of high Fe charge states
(with charges ¿ 13) is significantly enhanced during ICMEs. However, since
during normal solar wind conditions their contribution to the elemental
density can be neglected, these high Fe charge states mainly contribute to the
Fe elemental abundance for very high O charge state ratios, that is
$\log\left(\frac{n^{O7+}}{n^{O6+}}\right)>-0.5$. Thus, for corresponding slow
solar wind with $\log\left(\frac{n^{O7+}}{n^{O6+}}\right)>-0.5$, Figure 1 is
probably not representative. It is interesting to note that typically ICMEs
are also associated with very high FIP ratios, whereas here the Fe FIP ratio
is lower than expected for high O charge state ratios. Furthermore, for Mg and
Si, the same high O charge state ratios are shown in 2001–2006 as for Fe, but
the FIP ratios of Mg and Si nevertheless continues to increase with increasing
O charge state ratio. For each O charge state ratio bin, the distribution of
FIP ratios of Mg, Si, and Fe are asymmetric. Therefore, the median per O
charge state ratio bin, which is indicated with solid gray lines in Figure 1,
is not a good proxy for the most frequently observated FIP ratios, but instead
always overestimate them.

In the upper panels in Figure 2, the yearly median of the FIP ratio for Mg,
Si, and Fe is shown. In the lower panels, the number of samples per bin are
given for each year. For all three low FIP elements, the median FIP ratio is
lower in the years 2001–2006 and higher in the years 2007–2010. This indicates
a solar cycle dependence of the FIP effect for all three elements. However,
this effect is small compared to the variability of the observations as
indicated by the percentiles in Figure 2. Unlike the most frequently observed
Fe FIP ratios, the median of the Fe FIP ratios behaves more like the
respective yearly medians of the Mg and Si FIP ratios. Nevertheless, the slope
of the median Fe FIP ratio for high O charge ratios is smaller during the
solar activity maximum than during the solar activity minimum. This is not the
case for the year 2003, probably because of the unusually high fraction of
coronal hole wind during this year (see, e.g., Figure 2 in Pilleri et al.
2015).

Figure 3 compares the FIP ratios of each pair of low FIP elements (Mg versus
Fe, Si versus Fe, and Mg versus Si) for each year. With the help of
statistical tests, for each O charge state ratio bin we investigated whether
the distributions of FIP ratios of each pair of low FIP elements are likely to
follow the same or different distributions. The shaded/hatched areas mark
significant differences between the two respective mean values (gray shading,
$p<0.05$ according to the Kolmogorov–Smirnoff test; Kolmogorov 1933; Smirnov
1948) and empirical standard deviations (cross-hatching, $p<0.05$ based on
Levene’s test; Olkin 1960). Both statistical tests are suitable for non-normal
distributions and were only applied if a minimum of 100 samples were observed
per O charge state ratio bin. Both the mean and the standard deviation of Mg
and Si are significantly different for all years and over almost the complete
range of O charge state ratios. The distributions of Si and Fe are
significantly different as well for intermediate O charge state ratio in all
years, but show more similarities both for low and high O charge state ratios.
For Fe and Mg, the mean of the respective distribution is also significantly
different for most O charge state ratio bins, but for most years the empirical
standard deviations of the respective distributions are comparable for
intermediate and low O charge state ratios (which correspond mainly to the
coronal hole wind). Except for 2002, for high O charge state ratios (which
correspond mainly to the slow solar wind), and thus for the regime that
exhibits the unexpected behavior in the Fe FIP ratio, both mean and empirical
standard deviation of Fe and Mg differ significantly. It is also notable that
the bulk of the ratio distributions is not at one (except for high O charge
state ratio bins in the cases of Mg and Fe). In any O charge state ratio bin,
there is on average less Si observed than Fe, and for most O charge state
ratio bins more Mg than Fe. This is consistent with the observations in Reames
(2018) who observed higher Mg than Fe abundances for co-rotating interaction
regions, the inter stream solar wind, and the coronal hole wind. However,
Pilleri et al. (2015) instead found higher abundances for Fe than for Mg. We
suspect that the difference arises from a respective omission or inclusion of
Ca ions in the fitting process. Including Ca probably leads to some counts
that were caused by Fe ions to be incorrectly assigned to Ca which leads to an
underestimated Fe abundance. Excluding Ca from the fitting process
correspondingly probably leads to incorrectly assigning counts caused by Ca
ions to Fe which leads to an overestimation of Fe. One or both effects can
result in the different observed relative abundances. However, in all cases
these differences of the mean FIP ratio are within the respective error bars.

Although the distribution of the Mg and Si FIP ratios in Figure 1 appear
similar, Figure 3 shows that for most O charge state ratio bins, the
respective distributions of Mg and Si have significantly different means and
standard deviations.

To ensure that the effect discussed here is not simply caused by different
mixtures of different solar wind types in each year, in the following we sort
the observations by solar wind type. Figure 4 and Figure 5 show only solar
wind from 2001–2006 and 2007–2010, respectively, but the observations are also
separated by solar wind type according to the Xu & Borovsky (2015) scheme.
During 2001–2006 for the coronal hole wind, all three elements show the same
qualitative increase in the FIP ratio with increasing O charge state ratio;
however, the FIP effect is most pronounced for Mg and weakest for Si. Even for
the coronal hole wind, the Fe FIP ratio distribution appears broader than for
Mg and Si during this time period. For the slow solar wind, in particular for
helmet streamer plasma, Fe shows a qualitatively different behavior from that
of Mg and Si with increasing O charge state ratio during 2001–2006. For Fe,
sector reversal plasma exhibits a particularly broad FIP ratio distribution
and no clear dependence of the FIP effect on the O charge state ratio. During
2007–2010, the distribution of the observed Fe FIP ratios is also broader than
for Mg and Si, but for each solar wind type the qualitative behavior is more
similar.

##  4 Discussion and conclusion

The observations reveal that the long-term behavior of the elemental
composition is qualitatively different for Fe and for Mg and Si. While for Mg
and Si the FIP ratio always increases with increasing O charge state ratio in
all considered years and for all solar wind types, this is not the case for
Fe. For Fe, the FIP ratio reaches a plateau or broad peak for logarithms of
the O charge state ratio between 0.1 and 1 during the years 2001–2006. As also
supported by Figure 4 and Figure 5, the differences are most pronounced for
slow solar wind. That the Fe FIP ratio behaves differently from Mg and Si
mainly in the slow solar wind during the declining phase of the solar activity
maximum can indicate that a different fractionation or release mechanism is
dominant during this period than during the solar activity minimum.

In addition, the Fe FIP distribution per O charge state ratio bin is typically
broader than the respective distributions for Mg and Si. Based on the
available models of the FIP effect, such a qualitative difference between the
FIP ratios of low FIP elements is not expected. Thus, we believe that a full
explanation of the observations discussed here requires refinement and
improvement of the available models of the FIP effect.

The observations presented here clearly indicate that the assumption that all
low FIP elements show a highly correlated FIP effect is not always valid, and
thus that one or more other fractionation mechanisms play an important role.

###### Acknowledgements.

Part of this work was supported by the Deutsche Forschungsgemeinschaft (DFG)
project number Wi-2139/11-1 . We thank the referee, Martin Laming, for his
insightful, helpful, and detailed remarks. We also thank the science teams of
ACE/SWEPAM, ACE/MAG, as well as ACE/SWICS for providing the respective level 2
and level 1 data products.

## References

  * Arge & Mullan (1998) Arge, C. & Mullan, D. 1998, Solar Physics, 182, 293
  * Baker et al. (2015) Baker, D., Brooks, D., Démoulin, P., et al. 2015, The Astrophysical Journal, 802, 104
  * Benenson et al. (2006) Benenson, W., Harris, J. W., Stöcker, H., & Lutz, H. 2006, Handbook of physics (Springer Science & Business Media)
  * Berger (2008) Berger, L. 2008, PhD thesis, Kiel, Christian-Albrechts-Universität, Diss., 2008
  * Bochsler et al. (2000) Bochsler, P., Ipavich, F. M., Paquette, J. A., Weygand, J. M., & Wurz, P. 2000, Journal of Geophysical Research, 105, 12
  * Dahlburg et al. (2016) Dahlburg, R. B., Laming, J. M., Taylor, B., & Obenschain, K. 2016, The Astrophysical Journal, 831, 160
  * Del Zanna & Mason (2003) Del Zanna, G. & Mason, H. 2003, Astronomy & Astrophysics, 406, 1089
  * Doschek et al. (2015) Doschek, G., Warren, H., & Feldman, U. 2015, The Astrophysical Journal Letters, 808, L7
  * Feldman (1992) Feldman, U. 1992, Physica Scripta, 46, 202
  * Feldman & Widing (2003) Feldman, U. & Widing, K. G. 2003, Space Science Reviews, 107, 665
  * Gloeckler et al. (1998) Gloeckler, G., Cain, J., Ipavich, F., et al. 1998, in The Advanced Composition Explorer Mission (Springer), 497–539
  * Grevesse & Sauval (1998) Grevesse, N. & Sauval, A. 1998, Space Science Reviews, 85, 161
  * Jian et al. (2011) Jian, L., Russell, C., & Luhmann, J. 2011, Solar Physics, 274, 321
  * Jian et al. (2006) Jian, L., Russell, C., Luhmann, J., & Skoug, R. 2006, Solar Physics, 239, 393
  * Klecker et al. (2009) Klecker, B., Möbius, E., Popecki, M., et al. 2009, in Proceeding 31th International Cosmic Ray Conference
  * Kolmogorov (1933) Kolmogorov, A. 1933, Inst. Ital. Attuari, Giorn., 4, 83
  * Laming (2012) Laming, J. M. 2012, The Astrophysical Journal, 744, 115
  * Laming (2015) Laming, J. M. 2015, Living Reviews in Solar Physics, 12, 1
  * Laming (2017) Laming, J. M. 2017, The Astrophysical Journal, 844, 153
  * Laming et al. (2017) Laming, J. M., Heber, V. S., Burnett, D. S., et al. 2017, The Astrophysical Journal Letters, 851, L12
  * Marsch et al. (1995) Marsch, E., von Steiger, R., & Bochsler, P. 1995, Astronomy and Astrophysics, 301, 261
  * McComas et al. (1998) McComas, D., Bame, S., Barker, P., et al. 1998, in The Advanced Composition Explorer Mission (Springer), 563–612
  * Meyer (1985) Meyer, J.-P. 1985, The Astrophysical Journal Supplement Series, 57, 173
  * Müller et al. (2013) Müller, D., Marsden, R. G., Cyr, O. S., Gilbert, H. R., et al. 2013, Solar Physics, 285, 25
  * Olkin (1960) Olkin, I. 1960, Contributions to probability and statistics: essays in honor of Harold Hotelling (Stanford University Press)
  * Peter (1996) Peter, H. 1996, Astronomy and Astrophysics, 312, L37
  * Peter (1998) Peter, H. 1998, Astronomy and Astrophysics, 335, 691
  * Pilleri et al. (2015) Pilleri, P., Reisenfeld, D. B., Zurbuchen, T., et al. 2015, The Astrophysical Journal, 812, 1
  * Pucci et al. (2010) Pucci, S., Lie-Svendsen, Ø., & Esser, R. 2010, The Astrophysical Journal, 709, 993
  * Reames (2018) Reames, D. V. 2018, Solar Physics, 293, 47
  * Richardson & Cane (2010) Richardson, I. & Cane, H. 2010, Solar Physics, 264, 189
  * Schmelz et al. (2012) Schmelz, J. T., Reames, D. V., von Steiger, R., & Basu, S. 2012, The Astrophysical Journal, 755, 33
  * Schwadron (1999) Schwadron, N. 1999, Astrophysical Journal, 521, 868
  * Smirnov (1948) Smirnov, N. 1948, The annals of mathematical statistics, 19, 279
  * Smith et al. (1998) Smith, C. W., L’Heureux, J., Ness, N. F., et al. 1998, in The Advanced Composition Explorer Mission (Springer), 613–632
  * von Steiger & Geiss (1989) von Steiger, R. & Geiss, J. 1989, Astron. Astrophys., 225, 222
  * von Steiger & Zurbuchen (2015) von Steiger, R. & Zurbuchen, T. 2015, The Astrophysical Journal, 816, 13
  * Xu & Borovsky (2015) Xu, F. & Borovsky, J. E. 2015, Journal of Geophysical Research: Space Physics, 120, 70
  * Zhao & Fisk (2010) Zhao, L. & Fisk, L. 2010, in SOHO-23: Understanding a Peculiar Solar Minimum, Vol. 428, 229