1Vanderbilt University, Nashville, TN;2Vanderbilt University Medical Center,
Nashville, TN;
3TripleBlind, Kansas City, MO;4UTHealth, Houston, TX

# Split Learning for Distributed Collaborative
Training of Deep Learning Models in Health Informatics

Zhuohang Li  MS1 Chao Yan  PhD2 Xinmeng Zhang  BS1 Gharib Gharibi  PhD3
Zhijun Yin  PhD1,2 Xiaoqian Jiang  PhD4 Bradley A. Malin  PhD1,2

## Abstract

Deep learning continues to rapidly evolve and is now demonstrating remarkable
potential for numerous medical prediction tasks. However, realizing deep
learning models that generalize across healthcare organizations is
challenging. This is due, in part, to the inherent siloed nature of these
organizations and patient privacy requirements. To address this problem, we
illustrate how split learning can enable collaborative training of deep
learning models across disparate and privately maintained health datasets,
while keeping the original records and model parameters private. We introduce
a new privacy-preserving distributed learning framework that offers a higher
level of privacy compared to conventional federated learning. We use several
biomedical imaging and electronic health record (EHR) datasets to show that
deep learning models trained via split learning can achieve highly similar
performance to their centralized and federated counterparts while greatly
improving computational efficiency and reducing privacy risks.

## Introduction

Recent advances in deep learning algorithms have enabled the development of
neural networks with promising performance for a variety of healthcare data
types that were previously considered challenging with traditional statistical
methods, including medical images 1, natural languages 2, and structured
electronic health records (EHR) 3. Despite the remarkable progress that has
been achieved, most current deep learning models in the healthcare domain are
developed using data from only a single site. Yet it is evident that no single
healthcare organization can collect a sufficient amount of data on diverse
populations, as well as variations in organizational practices, to represent
the distribution of the general patient population. Consequently, models
developed from single-site data oftentimes lack sufficient generalizability
and may perform poorly when applied to other sites 4. To resolve this problem,
one natural solution would be to enable disparate healthcare organizations to
collaboratively train a model. However, such collaborations have met numerous
obstacles, ranging from concerns over data rights to patient privacy.

Over the past several years, distributed learning has been investigated as a
strategy to enable multiple data holders to contribute to the development of
deep learning models while protecting the privacy of the underlying raw
records. Specifically, one of the most popular variants of distributed
learning is federated learning (FL) 5; 6, which enables data holders to
contribute to the training of a learning model under the orchestration of a
central server by exchanging only focused model updates while maintaining
private data locally. The ability to maintain the privacy of record-level data
has drawn particular research interest from the healthcare community 7; 8.
Still, the canonical form of federated learning requires model builders to
reveal the details about their local models (i.e., model architecture and
model parameters) and such information can be leveraged to make inferences
about the privately maintained local data records. As a result, in
untrustworthy environments, federated learning needs to be jointly implemented
in concert with additional mechanisms to enhance its privacy support. There
are various approaches for doing so; some of the popular methods include 1)
differential privacy (DP) 9; 10 which leverages a randomization mechanism
(e.g., additive Laplacian noise) to provide privacy guarantees for individual
records for algorithms on aggregate databases, 2) secure multiparty
computation (SMC) 11; 12 a cryptographic solution to enable a set of parties
to compute a joint function on their private data without revealing anything
but the prescribed output and 3) homomorphic encryption (HE) 13; 14, which
allows a party to compute certain mathematical operations on ciphertexts
without decrypting them. However, in practice, these additional privacy
protection measures often come at the cost of either significantly harming
model utility (e.g., predictive performance) or increasing computational
complexity, which leads to long runtimes and costly computation in cloud
computing environments.

In this paper, we investigate split learning (SL) as a new paradigm for multi-
institutional collaborative learning of deep neural networks across
distributed health data. Similar to conventional distributed learning, the
split learning framework is composed of a server (i.e., the coordinator) and
multiple healthcare organizations (i.e., the data contributors) 15. Its
uniqueness is that under the split learning setting, the deep neural network,
also referred to as the global model, is divided into two sub-models (i.e.,
the client model and the server model) according to a specific layer known as
the cut layer 16. The healthcare organizations and the server hold only their
portion of the model, which they do not share with each other. At each
training round, the healthcare organizations only train the first part of a
deep neural network and send, what the literature has called, the smashed data
(i.e., the latent representations of raw data derived from the client model)
to the server. The server then completes the rest of the forward propagation
and computes the loss function without accessing the clients’ raw data.
Finally, the training round is concluded with a backward pass, where the
server sends back to the clients the computed gradients, which are then used
to update the client model. This process is equivalent to a training epoch in
the centralized learning setting and will iterate until the global model
converges.

Table 1 provides a qualitative comparison between different distributed
learning frameworks. Notably, as is in the federated learning setting, no
sensitive raw data is shared during the split learning training process, which
maintains the privacy of patients’ data. Moreover, in split learning, neither
the server nor the clients have complete knowledge of the global model’s
architecture and weights. This is a major difference from federated learning,
where the server has full access to the client’s model. This notion of
incompleteness in knowledge about the global model, combined with the minimal
information encoded in the smashed data, further reduces the risk of privacy
leakage during training. This is also beneficial to the server in scenarios
where the server hopes not to reveal its developed model architecture, which
may be considered proprietary. In addition to privacy benefits, split learning
can also alleviate the computational burden on the healthcare organizations’
side by offloading part of the training process of the deep neural network to
the server (e.g., a data center) which typically has access to more
computational power at a cheaper rate. In addition to the qualitative study,
we further 1) provide a quantitative analysis of the split learning framework
by conducting experiments across three biomedical image datasets (PathMNIST
17, OrganAMNIST 18, and BloodMNIST 19) and two EHR datasets (eICU 20 and a
private dataset from Vanderbilt University Medical Center 21), 2) perform an
analysis to compare the privacy risk under the split learning and federated
learning framework, and 3) investigate the trade-off between privacy, model
utility, and client-side model training efficiency in split learning. Our
results suggest that split learning can consistently achieve comparable
performance as federated learning while providing enhanced privacy and
computational efficiency for the participating health organizations.

Table 1: A qualitative comparison of different distributed learning schemes.

| Privacy Implications |  |
---|---|---|---
Framework |  | Protection over
---
Raw Data
| Protection over
---
Model Parameters
| Protection over
---
Model Architecture
Model Utility | Computational Efficiency
FL 5 | Yes | No | No | High | Moderate
FL+DP 10 | Yes | Variable∗ | No | Variable∗ | Moderate
FL+SMC 12 | Yes | No | No | High | Low
FL+SMC+HE 14 | Yes | Yes | No | High | Low
SL 16 | Yes | Partial† | Partial† | High | High

  * •

∗ Depends on the choice of privacy parameters and it’s typically a trade-off
between privacy and utility.

  * •

† Associated with the relative location of the cut layer: only the shallow
layers of the model (up to the cut layer) are protected from the server.

## Method

#### Federated Learning.

Federated learning can be conducted among either a small group of
organizations (cross-silo) or a large population of mobile devices (cross-
device). In this paper, we focus on the cross-silo setting, since it is more
prevalent in the healthcare domain. As depicted in Figure1(a), we assume that
there are $k$ clients who participate in the model training. Each client holds
$n_{i}$ data samples ($i\in\\{1,2,...,k\\}$). $n=\sum_{i}^{k}n_{i}$ is the
total data size. The objective of the collaboration is to train a neural
network $F_{\mathbf{w}}$ parameterized by weights $\mathbf{w}$. At the
beginning of the model training, each client initializes its local model
$\mathbf{w}^{c}_{i}$ in parallel. For each training epoch, each client
calculates the loss
$\mathcal{L}\big{(}F_{\mathbf{w}^{c}_{i}}(\mathbf{X}_{i}),\mathbf{y}_{i}\big{)}$
using its own data $\mathbf{X}_{i}$ with labels $\mathbf{y}_{i}$ in parallel.
A local model update is computed through gradient descent and shared with the
server:
$\mathbf{w}^{c}_{i}\leftarrow\mathbf{w}^{c}_{i}-\eta\cdot\frac{\partial\mathcal{L}}{\partial\mathbf{w}^{c}_{i}}$.
Finally, at the end of each training epoch, the server updates the global
model weights by computing a weighted average of the local models, a process
known as federated averaging 5:
$\mathbf{w}\leftarrow\sum_{i}^{k}\frac{n_{i}}{n}\cdot\mathbf{w}^{c}_{i}$. This
process is repeated until the global model converges.

(a) Federated learning

(b) Split learning

Figure 1: Illustration of the federated learning and split learning
frameworks.

#### Split Learning.

As shown in Figure 1(b), in the split learning setting, a neural network $F$
is partitioned into two separate sub-models. The first is the client model
$h_{\mathbf{w}^{c}}$, which takes the raw data $\mathbf{X}$ and outputs latent
representations of the data known as the smashed data. The second is the
server model $f_{\mathbf{w}^{s}}$, which makes predictions based on the
smashed data, i.e., $F(\mathbf{X})=(f_{\mathbf{w}^{s}}\circ
h_{\mathbf{w}^{c}})(\mathbf{X})$. The clients and the server only have access
to their own part of the model and cannot access the other part.

Each training step of a neural network can be described as a forward pass
where the loss function is computed and a backward pass where the model
parameters are updated by back-propagating the error through gradients. In the
canonical version of split learning, at each forward pass, the client computes
the smashed data $h_{\mathbf{w}^{c}}(\mathbf{x})$ and then shares the smashed
data along with its label to the server. The server makes a prediction of the
smashed data using the server model to compute the loss
$\mathcal{L}\Big{(}f_{\mathbf{w}^{s}}\big{(}h_{\mathbf{w}^{c}}(\mathbf{X})\big{)},\mathbf{y}\Big{)}$.
In the backward pass, the server first updates its model according to
$\mathbf{w}^{s}\leftarrow\mathbf{w}^{s}-\eta\cdot\frac{\partial\mathcal{L}}{\partial\mathbf{w}^{s}}$
and sends the gradients of the smashed data
$\frac{\partial\mathcal{L}}{\partial h_{\mathbf{w}^{c}}(\mathbf{X})}$ to the
client. The client then computes the gradients of the client model
$\frac{\partial\mathcal{L}}{\partial\mathbf{w}^{c}}$ and updates the model
parameters accordingly i.e.,
$\mathbf{w}^{c}\leftarrow\mathbf{w}^{c}-\eta\cdot\frac{\partial\mathcal{L}}{\partial\mathbf{w}^{c}}$,
where $\eta$ is the learning rate. Finally, in the multiple clients’ scenario,
the client shares its model parameters with the next client where this process
repeats. Note that the raw data is never shared in the process and the server
only sees a compact representation of the client’s private data (the smashed
data). Moreover, the computational cost on the clients’ end is greatly reduced
since the client is only responsible for the computation of the first part of
the model. It should be recognized that there are variants of split learning
that do not require sharing labels or allow learning on vertically partitioned
data 16.

#### Speeding up Split Learning with Federation.

The canonical split learning framework processes each client in a sequential
order, which can become very time-consuming if either the number of clients or
the amount of data maintained by each client is large. Inspired by federated
learning, the split learning framework can be modified to enable clients to
compute their updates in a parallel manner and thereby accelerate the learning
process 22. Specifically, at each round of training, $k$ clients first compute
their smashed data in parallel
$h_{\mathbf{w}^{c}_{i}}(\mathbf{X}_{i}),i\in\\{1,2,...,k\\}$ and send the
results to the server. Next, the server computes the loss
$\mathcal{L}_{i}\Big{(}f_{\mathbf{w}^{s}_{i}}\big{(}h_{\mathbf{w}^{c}_{i}}(\mathbf{X}_{i})\big{)},\mathbf{y}_{i}\Big{)}$
and the gradients $\frac{\partial\mathcal{L}_{i}}{\partial\mathbf{w}^{c}_{i}}$
and sends back the gradients to each individual clients. The server then
updates its model by averaging the gradients from all clients:
$\mathbf{w}^{s}\leftarrow\mathbf{w}^{s}-\eta\sum_{i}^{k}\frac{n_{i}}{n}\cdot\frac{\partial\mathcal{L}_{i}}{\partial\mathbf{w}^{s}_{i}}$.
Similarly, the clients update their models via federated averaging, i.e.,
$\mathbf{w}^{c}\leftarrow\mathbf{w}^{c}-\eta\sum_{i}^{k}\frac{n_{i}}{n}\cdot\frac{\partial\mathcal{L}_{i}}{\partial\mathbf{w}^{c}_{i}}$,
with the help of a separate server referred to as the federated server. By
allowing parallel computing on both the clients’ and the server’s side, this
variant of split learning (named SplitFedv1 in 22) can significantly speed up
the training process. However, the drawback of this approach is that to
protect the client’s data privacy, it is required that the federated server is
a trustworthy third party that does not collude with the main server. In
addition, the federation process may negatively affect the utility of the
converged model. Nevertheless, a low computation latency is often desired for
deploying distributed learning solutions to large-scale real-world
applications. Thus, we use this variant of split learning as the default in
our experiments.

## Evaluation

### Experimental Setup

#### Datasets.

Table 2: A summary of the datasets used in this study.

Dataset | Data Type | Task | Label Distribution
---|---|---|---
PathMNIST | Colon pathology | Multi-class |
OrganAMNIST | Abdominal CT | Multi-class |
BloodMNIST | Blood cell microscope | Multi-class |
eICU | Electronic health record | Binary-class |
VUMC | Electronic health record | Binary-class |

We utilize five datasets to support evaluations in two types of settings -
biomedical image classification and clinical concept predictions from
structured EHR data. Specfically, we apply the following three datasets from
the MedMNIST 23 benchmark for biomedical image classification tasks: (1)
PathMNIST 17: a colon pathology dataset containing $100,000$ non-overlapping
image patches from hematoxylin & eosin stained histological images for
classifying $9$ types of tissues. We follow the recommended train/test split
in our experiment, resulting in a total number of $89,996$ images for
training. An additional $7,180$ image patches collected from a different
clinical center are reserved for testing; (2) OrganAMNIST 18: a 2D image
dataset cropped from the axial view of 3D computed tomography (CT) images from
Liver Tumor Segmentation Benchmark 18 for performing classification of $11$
body organs. The dataset is partitioned into two disjoint sets, with $34,581$
images for training and $17,778$ for testing; and (3) BloodMNIST 19: a
peripheral blood cell images dataset containing individual normal cells
organized into $8$ classes. The dataset is partitioned into training and
testing sets, each with $11,959$ and $3,421$ images, respectively. For the EHR
prediction tasks, we use the following two datasets: (1) eICU 20: a public EHR
dataset containing more than $140,000$ patients’ hospital visit records. The
task is to predict the risk of being readmitted to the ICU in the next $15$
days (i.e., binary classification) given the clinical activities during the
current ICU stay. Following previous studies 24; 25, we consider five types of
events as features, including diagnosis, lab test, medication, physical exam,
and treatment. We use non-overlapping training and testing sets containing
data from $30,000$ and $10,000$ patients, respectively; and (2) VUMC 21: a
private EHR dataset collected from Vanderbilt University Medical Center
containing all adult inpatient visits in 2019. The task is to predict whether
the patient will be discharged the next day to home or other care facilities.
Visits shorter than 24h or of patients who died during hospitalization are
excluded, resulting in a total number of $26,283$ patients with an average age
of $52.9$. Table 2 provides a summary of the description and label
distribution for each dataset.

Table 3: Model architecture for biomedical image classification.

Layer | Kernel | Stride | Output
---|---|---|---
Conv2D | $3\times 3$ | $1\times 1$ | $16$
BatchNorm2D | $-$ | $-$ | $16$
Conv2D | $3\times 3$ | $1\times 1$ | $16$
BatchNorm2D | $-$ | $-$ | $16$
MaxPool2D | $2\times 2$ | $2\times 2$ | $16$
1-4 Conv2D | $3\times 3$ | $1\times 1$ | $64$
BatchNorm2D | $-$ | $-$ | $64$
Conv2D | $3\times 3$ | $1\times 1$ | $64$
BatchNorm2D | $-$ | $-$ | $64$
Conv2D | $3\times 3$ | $1\times 1$ | $64$
BatchNorm2D | $-$ | $-$ | $64$
MaxPool2D | $2\times 2$ | $2\times 2$ | $64$
FC | $-$ | $-$ | $128$
FC | $-$ | $-$ | $128$
FC | $-$ | $-$ | # of classes

#### Deep Learning Models.

For biomedical image classification tasks, we rely on a convolutional neural
network containing five convolutional layers, two max-pooling layers, and
three fully-connected (FC) layers with ReLU activation. Table 3 provides the
details of the network architecture. The client owns the first part of the
model, containing two convolutional layers and one max-pooling layer. For the
readmission prediction task on the eICU dataset, we apply the same model
architecture as described in 25. The client model utilizes separate encoders
for mapping different types of events into a same-length embedding sequence,
which is then processed by a Transformer encoder. The server model is a two-
layer fully-connected network for making final predictions. For the discharge
prediction task with the VUMC dataset, we use a four-layer fully-connected
network ($2808-64-32-32-1$) with ReLU activation, where the model is split
after the first layer.

#### Centralized/Distributed Learning Setting.

In the centralized learning scenario, we train the deep learning model on the
entire training set for $50$ epochs using Adam optimizer with a batch size of
$256$. By default, we use a learning rate of $10^{-4}$ and a weight decay of
$10^{-5}$. Specially, for the readmission prediction task on the eICU dataset,
we set the learning rate to be $5\times 10^{-4}$. For the discharge prediction
task on the VUMC dataset, we use a weight decay of $10^{-4}$. In the case of
federated and split learning scenarios, we randomly partition the training
dataset into $k$ disjoint subsets and assign them to each client. The
parameters for the optimizers are set to be the same as in centralized
learning. Unless mentioned otherwise, we use a default number of clients $k=5$
in our experiments.

Table 4: A comparison of final performance measured on the test dataset for
centralized learning (CL), federated learning (FL), and split learning (SL)
(reported with a $95\%$ confidence interval).

| PathMNIST | OrganAMNIST | BloodMNIST | eICU | VUMC
---|---|---|---|---|---
Accuracy | AUROC | Accuracy | AUROC | Accuracy | AUROC | AUPRC | F1 | Kappa | AUPRC | F1 | Kappa
CL |  | 0.8346
---
$\pm$0.0088
| 0.9734
---
$\pm$0.0071
| 0.8773
---
$\pm$0.0051
| 0.9901
---
$\pm$0.0010
| 0.9379
---
$\pm$0.0022
| 0.9958
---
$\pm$0.0005
| 0.6389
---
$\pm$0.0127
| 0.6083
---
$\pm$0.0057
| 0.4735
---
$\pm$0.0087
| 0.7898
---
$\pm$0.0058
| 0.7147
---
$\pm$0.0037
| 0.6565
---
$\pm$0.0042
FL |  | 0.8225
---
$\pm$0.0175
| 0.9690
---
$\pm$0.0043
| 0.8689
---
$\pm$0.0065
| 0.9890
---
$\pm$0.0010
| 0.9165
---
$\pm$0.0107
| 0.9932
---
$\pm$0.0012
| 0.6384
---
$\pm$0.0122
| 0.5943
---
$\pm$0.0101
| 0.4675
---
$\pm$0.0098
| 0.7884
---
$\pm$0.0019
| 0.7122
---
$\pm$0.0029
| 0.6547
---
$\pm$0.0035
SL |  | 0.8127
---
$\pm$0.0129
| 0.9673
---
$\pm$0.0069
| 0.8538
---
$\pm$0.0067
| 0.9864
---
$\pm$0.0012
| 0.9066
---
$\pm$0.0096
| 0.9917
---
$\pm$0.0013
| 0.6331
---
$\pm$0.0109
| 0.6000
---
$\pm$0.0074
| 0.4693
---
$\pm$0.0078
| 0.7840
---
$\pm$0.0028
| 0.7116
---
$\pm$0.0026
| 0.6524
---
$\pm$0.0033

(a) PathMNIST

(b) OrganAMNIST

(c) BloodMNIST

(d) eICU

(e) VUMC

Figure 2: Comparison of convergence by measuring the per-epoch performance on
the test dataset for federated learning and split learning.

(a) PathMNIST

(b) OrganAMNIST

(c) BloodMNIST

(d) eICU

(e) VUMC

Figure 3: Linear regression results of the per-epoch performance of federated
learning vs. split leaning on $5$ datasets.

#### Metrics.

We use the following metrics for the biomedical image classification tasks:
(1) Accuracy: the average classification accuracy; and (2) AUROC: the average
area under the receiver operating characteristic (ROC) curves of each class
against the rest. For the EHR prediction tasks, we report the following
measures: (1) AUPRC: the average area under the precision-recall curve; (2)
F1: the average F1 score; and (3) Kappa: Cohen’s Kappa score, which measures
the agreement between two raters on a classification task, defined as
$\kappa=\frac{p_{o}-p_{e}}{1-p_{e}}$, where $p_{o}$ is the empirical
probability of agreement on the label assigned to any sample (i.e., the
observed agreement ratio), and $p_{e}$ is the expected agreement when both
raters assign labels randomly. The Kappa ranges from $-1$ to $1$, with a
larger value indicating a stronger agreement. For each setting in our
experiment, we train the neural network on the training dataset with five
different random seeds and report its performance as measured on the hold-out
testing dataset.

(a) Federated learning

(b) Split learning

(c) Difference

Figure 4: Sensitivity analysis on the VUMC dataset.

### Utility Analysis

#### Performance.

We conducted experiments using centralized learning, federated learning, and
split learning under the same conditions for $50$ epochs. To eliminate the
effects of overfitting in centralized learning, we assume early stopping is
employed and report the best performance measured on the test dataset. As
shown in Table 4, the centralized learning consistently achieved the highest
performance across all five datasets, which can be considered the upper bound
of the performance for distributed learning. While federated learning
outperformed split learning slightly on four datasets (excluding eICU), the
difference in performance between them was almost negligible (usually $<1\%$),
with margins of error being similar.

#### Convergence.

Figure 2 compares the convergence of the global model between federated
learning and split learning. The first and second rows plot the model
performance measured on the test dataset after each epoch for split and
federated learning, respectively. The third row plots the relative
approximation error between federated and split learning, defined as
$\delta=|\frac{v_{SL}-v_{FL}}{v_{FL}}|\times 100\%$, where $v_{SL}$ and
$v_{FL}$ denote the performance of split and federated learning model
respectively. We observe that except for the differences caused by the
initialization at the beginning of training, both split and federated learning
are able to converge at a similar rate. For some datasets (e.g., eICU and
VUMC), split learning can converge even faster than federated learning. This
observation was further verified by the linear regression analysis shown in
Figure 3, which demonstrates a high correlation for all datasets (typically
with estimated coefficients $<1.2$ and R2 $>0.9$).

#### Scalibility.

To investigate the scalability of federated learning and split learning, we
perform a sensitivity analysis on the VUMC dataset with different numbers of
participating clients as well as different numbers of training data samples
per client. As shown in Figure 4, a similar pattern is observed for both
federated learning and split learning, where the performance of the resulting
model (measured by AUPRC) continues to improve as the number of clients and
the number of samples per client increases. Moreover, the differences in
performance between federated learning and split learning as shown in Figure
4(c) (green for positive and orange for negative values) are negligible. This
indicates that federated learning and split learning are both scalable to more
participating clients and a larger amount of training data.

#### Efficiency.

Table 5: Comparison on client-side model efficiency.

Dataset | Split Learning | Federated Learning
---|---|---
# of params | FLOPs | # of params | FLOPs
PathMNIST | 2,832 | 1,726,208 | 235,225 | 7,591,817
OrganAMNIST | 2,544 | 1,531,520 | 235,195 | 7,397,387
BloodMNIST | 2,832 | 1,726,208 | 235,096 | 7,591,688
eICU | 1,556,475 | $-^{*}$ | 1,558,556 | $-^{*}$
VUMC | 179,776 | 179,904 | 186,049 | 186,369

  * •

∗ Cannot be estimated since the input is heterogeneous clinical sequences with
variable lengths and diverse features.

Table 5 compares the efficiency on the client’s end in terms of model
parameters and the estimated floating point operations (FLOPs) required for
computing a model forward pass. Our results indicate that split learning
consumes less memory and requires fewer computations on the client side
compared to federated learning across all five datasets. Notably, split
learning outperforms federated learning significantly in medical image
recognition tasks, reducing memory usage by approximately $99\%$ and
computational requirements by $77\%$. However, the benefits of split learning
in EHR tasks are marginal, likely due to the complexity of prediction tasks,
the specific model architecture, and the choice of cut layer position used in
our experiments.

### Privacy Analysis

Both federated learning and split learning mitigate the systemic privacy risks
from traditional centralized learning by embedding a data minimization
principle in their design. Federated learning achieves this principle by
aggregating information collected from multiple data records into a focused
model update. By contrast, split learning limits the amount of collected
information about each data record by only sharing its smashed data.

For the simplicity of analysis, herein we assume that the number of dimensions
being revealed is proportional to the privacy risk, i.e., the amount of
private information being leaked. We note that this simplified privacy model
is by no means a rigorous measure of data privacy, but it can serve as a
baseline with room for refinement in the future. More specifically, let us
consider a client who contributes $n_{c}$ private data samples to participate
in training. Suppose the federated learning model has $N_{w}$ the total number
of parameters (i.e., $w\in\mathbb{R}^{N_{w}}$) and the cut layer size/smashed
data dimension is $d$ (i.e., $h_{w^{c}}(x)\in\mathbb{R}^{d}$). Then during
each training round, the average number of dimensions revealed to the server
per each training sample is $\frac{N_{w}}{n_{c}}$ for federated learning and
$d$ for split learning, as illustrated in Figure 5(a). Since the model update
is for a fixed number of dimensions (i.e., the same dimension as the model
parameters), federated learning benefits from having a large size of local
training data to average out the privacy risk. By contrast, the amount of
information revealed for each data sample in split learning is invariant to
the local data size and only dependent on the client’s model architecture
(i.e., the cut layer size). Since deeper layers tend to produce more compact
representations (i.e., smaller $d$), it is usually a trade-off between
computational efficiency and privacy for the client model in split learning.
As such, federated learning appears to be more suitable for scenarios where
every client possesses a sufficiently large quantity of data samples
($\geq\frac{N_{w}}{d}$) but discourages clients with fewer data from
participating due to the higher privacy risk. In our setting, the minimum
reasonable data size for federated learning ($\frac{N_{w}}{d}$) is $102$,
$2907$, and $12176$ for the biomedical image datasets, the VUMC dataset, and
the eICU dataset, respectively. However, since modern deep neural networks are
experiencing exponential growth in size, the minimum required data size can
easily scale up to a number that is very difficult to achieve in practice. For
example, if we train a ResNet-50 model 26 that contains over $23$ million
trainable parameters to recognize medical images and choose to split after the
adaptive pooling layer (corresponds to a cut layer size of $512$), we would
need at least over $44000$ data records from every site to justify the
adoption of federated learning. Such a requirement is oftentimes unattainable
for healthcare applications, where a client with only a few patient records
can also make a crucial contribution to the model. For instance, specialty
healthcare facilities (e.g., smaller oncology practices) may have much fewer
patients compared to general hospitals, but they typically have a much more
focused dataset in terms of certain diseases. Therefore, split learning
becomes more desirable in these circumstances as it is more versatile and can
provide the same level of privacy benefits regardless of the size of the
client’s local data.

(a) Privacy budget per sample measured by dimensionality

(b) Privacy protection scheme: information within the blue dashed line are
kept secret from the server

Figure 5: Conceptual comparison of privacy preservation in federated learning
(FL) vs. split learning (SL).

#### Risk of Inversion.

Compared to federated learning, split learning further reduces the risks of
model/gradient inversion attack 27; 28; 29; 30 by restricting model access for
both parties. As is depicted in Figure 5(b), learning can be seen as computing
a function $F$ on the client’s private data $X$ to get result $M$. In
federated learning, although the raw data $X$ is not directly accessible to
the server, the computation function $F$ and the result $M$ are both known to
the server, which leaves the opportunity for the server to approximate an
inverse function to infer $X$ from $M$ and break patient privacy. Differently,
split learning separates the function $F$ (i.e., the model) into the client’s
part $F_{C}$ and the server’s part $F_{S}$ and only the latter is revealed to
the server. Thus, the server in split learning only has imperfect information
about the computing process, which makes it quite challenging to compute the
exact inverse function.

## Discussion

#### Design Trade-offs.

There exist two major trade-offs in designing a deep learning model for split
learning. (1) Privacy-utility trade-off: Split learning reflects the data
minimization principle by allowing clients to extract and send only the
relevant features of their local data to the server, while keeping the rest of
the data private. The client model serves as a compressor which helps to
reduce the amount of information that is transmitted over the network while
preserving the most important information for the task at hand. Using a small
cut layer size further suppresses the amount of information in the smashed
data, but may negatively affect the model performance. Thus, one of the main
objectives of designing a split learning model is to choose a proper cut layer
size to find a balance between keeping enough information to capture the
important aspects of the data and reducing the amount of information to
minimize the privacy risk of releasing the smashed data. (2) Privacy-
efficiency trade-off: the information bottleneck theory 31 suggests that the
output of deeper layers in a neural network contains more information about
the task label and less information about the original input. As such,
choosing to split at a deeper layer may result in less redundant information
in the smashed data, thereby reducing the privacy risk. However, setting a
shallow layer as the cut layer allows the client to have a more compact model
and can thus give full scope to the advantages of split learning to reduce the
memory consumption and computational burden on the client side. Overall, these
tuning knobs in split learning offer the user great flexibility in terms of
configuring the privacy-utility-efficiency trade-off for specific tasks,
whereas federated learning lacks such customizability.

#### Limitations.

Despite the merits of split learning, there are several limitations we would
like to highlight. First, the current split learning framework only supports
decentralized training of deep neural networks but does not support the
training of traditional machine learning models, such as tree-based models.
Second, although split learning largely relieves the computational cost on the
client side, the communication cost is increased compared to federated
learning, as the client needs to communicate more frequently with the server
for every mini-batch to perform gradient descent. To address this, recent
studies propose to utilize an intermediate edge server 32, asynchronous
training scheme 33, or an automated software framework 34 to reduce the
communication overheads in split learning. Additionally, service providers who
do not wish to claim intellectual property over the developed model can share
the server-side model with the clients after training to eliminate the
communication overheads at inference time. Third, the privacy benefits of
split learning heavily rely on the inaccessibility of the client model and
thus might be vulnerable under a stronger threat model. For instance, in the
insider attack scenario where the server is able to collude with one of the
clients, the privacy protection over other clients’ data would be greatly
diminished.

#### Future Directions.

There are several notable problems that should be considered as future
research. First, how can we combine split learning with rigorous statistical
privacy methods (e.g., differential privacy) or secure multiparty computation
to achieve more rigorous privacy protection? Moreover, how can this be done
with minimal sacrifice to model utility or significantly increasing the
computational cost? Second, how can we design a more nuanced framework for
privacy in split learning? Specifically, we need to quantify the amount of
leaked private information via smashed data and the potential privacy risks.
Third, can we design a systematic framework to decide what the optimal cut
layer size and location for a given neural network is to minimize the privacy
risk while preserving most model utility? Fourth, how can we realize split
learning on heterogeneous data types and varying data distributions?

### Conclusion

A lack of sufficient data to cover the distribution of the general population
is one of the major impediments to developing practical deep learning models
for healthcare applications. In this work, we introduced split learning as a
new distributed learning paradigm for enabling multi-institutional
collaborative development of deep learning models across data silos without
accessing raw patient data. Through both in-depth qualitative analysis as well
as systematic quantitative experiments on five health datasets, we illustrated
that split learning can achieve similar model utility as federated learning
while providing better client-side model efficiency, lower risk of inversion,
enhanced protection over the model, and more flexible privacy protection over
clients’ data. Our findings suggest that split learning is a promising
alternative to federated learning for developing deep learning models without
violating the privacy of the data contributors in many healthcare tasks.

##### Acknowledgments

This research was sponsored in part by grant U54HG012510, the NIH Bridge2AI
Center. BM has been a paid consultant to TripleBlind AI, but for work
unrelated to this investigation.

## References

  * 1 Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, et al.  Dermatologist-level classification of skin cancer with deep neural networks.  Nature. 2017;542(7639):115-8.
  * 2 Wen A, Fu S, Moon S, El Wazir M, Rosenbaum A, Kaggal VC, et al.  Desiderata for delivering NLP to accelerate healthcare AI advancement and a Mayo Clinic NLP-as-a-service implementation.  NPJ Digit Med. 2019;2(1):130.
  * 3 Landi I, Glicksberg BS, Lee HC, Cherng S, Landi G, Danieletto M, et al.  Deep representation learning of electronic health records to unlock patient stratification at scale.  NPJ Digit Med. 2020;3(1):96.
  * 4 Cohen JP, Cao T, Viviano JD, Huang CW, Fralick M, Ghassemi M, et al.  Problems in the deployment of machine-learned models in health care.  CMAJ. 2021;193(35):E1391-4.
  * 5 McMahan B, Moore E, Ramage D, Hampson S, y Arcas BA.  Communication-efficient learning of deep networks from decentralized data.  In: Artificial intelligence and statistics. PMLR; 2017. p. 1273-82.
  * 6 Rieke N, Hancox J, Li W, Milletari F, Roth HR, Albarqouni S, et al.  The future of digital health with federated learning.  NPJ Digit Med. 2020;3(1):119.
  * 7 Kaissis GA, Makowski MR, Rückert D, Braren RF.  Secure, privacy-preserving and federated machine learning in medical imaging.  Nature Machine Intelligence. 2020;2(6):305-11.
  * 8 Dayan I, Roth HR, Zhong A, Harouni A, Gentili A, Abidin AZ, et al.  Federated learning for predicting clinical outcomes in patients with COVID-19.  Nat Med. 2021;27(10):1735-43.
  * 9 Abadi M, Chu A, Goodfellow I, McMahan HB, Mironov I, Talwar K, et al.  Deep learning with differential privacy.  In: Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security; 2016. p. 308-18.
  * 10 Truex S, Liu L, Chow KH, Gursoy ME, Wei W.  LDP-Fed: Federated learning with local differential privacy.  In: Proceedings of the Third ACM International Workshop on Edge Systems, Analytics and Networking; 2020. p. 61-6.
  * 11 Constable SD, Tang Y, Wang S, Jiang X, Chapin S.  Privacy-preserving GWAS analysis on federated genomic datasets.  In: BMC Med Inform Decis Mak. vol. 15. BioMed Central; 2015. p. 1-9.
  * 12 Bonawitz K, Ivanov V, Kreuter B, Marcedone A, McMahan HB, Patel S, et al.  Practical secure aggregation for privacy-preserving machine learning.  In: proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security; 2017. p. 1175-91.
  * 13 Froelicher D, Troncoso-Pastoriza JR, Raisaro JL, Cuendet MA, Sousa JS, Cho H, et al.  Truly privacy-preserving federated analytics for precision medicine with multiparty homomorphic encryption.  Nat Commun. 2021;12(1):5910.
  * 14 Hardy S, Henecka W, Ivey-Law H, Nock R, Patrini G, Smith G, et al.  Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryption.  arXiv preprint arXiv:171110677. 2017.
  * 15 Gupta O, Raskar R.  Distributed learning of deep neural network over multiple agents.  Journal of Network and Computer Applications. 2018;116:1-8.
  * 16 Vepakomma P, Gupta O, Swedish T, Raskar R.  Split learning for health: Distributed deep learning without sharing raw patient data.  arXiv preprint arXiv:181200564. 2018.
  * 17 Kather JN, Krisam J, Charoentong P, Luedde T, Herpel E, Weis CA, et al.  Predicting survival from colorectal cancer histology slides using deep learning: A retrospective multicenter study.  PLoS Med. 2019;16(1):e1002730.
  * 18 Bilic P, Christ PF, Vorontsov E, Chlebus G, Chen H, Dou Q, et al.  The liver tumor segmentation benchmark (lits). arXiv.  arXiv preprint arXiv:190104056. 2019.
  * 19 Acevedo A, Merino A, Alférez S, Molina Á, Boldú L, Rodellar J.  A dataset of microscopic peripheral blood cell images for development of automatic recognition systems.  Data in Brief. 2020;30.
  * 20 Pollard TJ, Johnson AE, Raffa JD, Celi LA, Mark RG, Badawi O.  The eICU Collaborative Research Database, a freely available multi-center database for critical care research.  Sci Data. 2018;5(1):1-13.
  * 21 Zhang X, Yan C, Malin BA, Patel MB, Chen Y.  Predicting next-day discharge via electronic health record access logs.  Journal of the American Medical Informatics Association. 2021;28(12):2670-80.
  * 22 Thapa C, Arachchige PCM, Camtepe S, Sun L.  Splitfed: When federated learning meets split learning.  In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 36; 2022. p. 8485-93.
  * 23 Yang J, Shi R, Wei D, Liu Z, Zhao L, Ke B, et al.  MedMNIST v2-A large-scale lightweight benchmark for 2D and 3D biomedical image classification.  Scientific Data. 2023;10(1):41.
  * 24 Choi E, Xu Z, Li Y, Dusenberry M, Flores G, Xue E, et al.  Learning the graphical structure of electronic health records with graph convolutional transformer.  In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34; 2020. p. 606-13.
  * 25 Yang C, Westover MB, Sun J.  ManyDG: Many-domain Generalization for Healthcare Applications.  In: Proceedings of the International Conference on Learning Representations; 2023. .
  * 26 He K, Zhang X, Ren S, Sun J.  Deep residual learning for image recognition.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition; 2016. p. 770-8.
  * 27 Fredrikson M, Jha S, Ristenpart T.  Model inversion attacks that exploit confidence information and basic countermeasures.  In: Proceedings of the 22nd ACM SIGSAC conference on computer and communications security; 2015. p. 1322-33.
  * 28 Zhang Y, Jia R, Pei H, Wang W, Li B, Song D.  The secret revealer: Generative model-inversion attacks against deep neural networks.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2020. p. 253-61.
  * 29 Zhu L, Liu Z, Han S.  Deep leakage from gradients.  Advances in neural information processing systems. 2019;32.
  * 30 Li Z, Zhang J, Liu L, Liu J.  Auditing privacy defenses in federated learning via generative gradient leakage.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition; 2022. p. 10132-42.
  * 31 Tishby N, Zaslavsky N.  Deep learning and the information bottleneck principle.  In: IEEE Information Theory Workshop. IEEE; 2015. p. 1-5.
  * 32 Turina V, Zhang Z, Esposito F, Matta I.  Federated or split? a performance and privacy analysis of hybrid split and federated learning architectures.  In: Proceedings of the IEEE 14th International Conference on Cloud Computing. IEEE; 2021. p. 250-60.
  * 33 Duan Q, Hu S, Deng R, Lu Z.  Combined federated and split learning in edge computing for ubiquitous intelligence in internet of things: State-of-the-art and future directions.  Sensors. 2022;22(16):5983.
  * 34 Gharibi G, Patel R, Khan A, Gilkalaye BP, Vepakomma P, Raskar R, et al.  An automated framework for distributed deep learning–a tool demo.  In: Proceedings of the IEEE 42nd International Conference on Distributed Computing Systems. IEEE; 2022. p. 1302-5.