00A0

# Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed
Opportunity

Raman Dutt
The University of Edinburgh
Edinburgh, UK
<EMAIL_ADDRESS>Linus Ericsson
The University of Edinburgh
Edinburgh, UK
<EMAIL_ADDRESS>Pedro Sanchez
The University of Edinburgh
Edinburgh, UK
<EMAIL_ADDRESS>Sotirios A. Tsaftaris
The University of Edinburgh
Edinburgh, UK
<EMAIL_ADDRESS>Timothy Hospedales
The University of Edinburgh
Edinburgh, UK
<EMAIL_ADDRESS>
###### Abstract

We present a comprehensive evaluation of Parameter-Efficient Fine-Tuning
(PEFT) techniques for diverse medical image analysis tasks. PEFT is
increasingly exploited as a valuable approach for knowledge transfer from pre-
trained models in natural language processing, vision, speech, and cross-modal
tasks, such as vision-language and text-to-image generation. However, its
application in medical image analysis remains relatively unexplored. As
foundation models are increasingly exploited in the medical domain, it is
crucial to investigate and comparatively assess various strategies for
knowledge transfer that can bolster a range of downstream tasks. Our study,
the first of its kind (to the best of our knowledge), evaluates 16 distinct
PEFT methodologies proposed for convolutional and transformer-based networks,
focusing on image classification and text-to-image generation tasks across six
medical datasets ranging in size, modality, and complexity. Through a battery
of more than 600 controlled experiments, we demonstrate performance gains of
up to 22% under certain scenarios and demonstrate the efficacy of PEFT for
medical text-to-image generation. Further, we reveal the instances where PEFT
methods particularly dominate over conventional fine-tuning approaches by
studying their relationship with downstream data volume.

##  1 Introduction

Medical imaging technologies are revolutionizing modern medicine in various
domains, covering both clinical diagnosis and biomedical research [30]. AI
models have demonstrated diagnostic capabilities on par with human experts
across a range of imaging modalities, including radiology [45, 19],
dermatology [20], and ophthalmology [14, 65], among others. Such AI methods
offer the potential to enhance healthcare access and overall health outcomes
for society [67], making them highly significant. However, a critical obstacle
in achieving this vision is the need for large training datasets essential for
mainstream end-to-end supervised deep learning from scratch. This challenge
continues to grow as state-of-the-art deep architectures [18] become
increasingly large and consequently more data-hungry over time. Acquiring the
necessary volume of data is typically unattainable at the required scale due
to privacy restrictions and the long-tailed nature of medical conditions of
interest and prohibitively expensive to annotate [60].

To tackle the data bottleneck, an expanding body of medical AI research [4, 3,
33] has adopted the _pre-train fine-tune_ or _transfer learning_ approach,
where models undergo training in two stages: Pre-training is conducted on
either out-of-domain non-medical images or unlabeled medical images, which are
easier to acquire, allowing this stage to scale to larger datasets. Fine-
tuning is then performed on in-domain medical images for the specific task.
The assumption is that parameters are mostly set during pre-training and only
require minimal fine-tuning for the target task. This approach has become
increasingly important since the emergence of ‘foundation models’ [8] —
powerful off-the-shelf pre-trained models — due to their use of large
architectures, extensive pre-training datasets, and massive training
computations. Several recent evaluations have used larger external datasets in
this manner to significantly enhance performance on medical tasks of interest
[4, 3, 33].

Within the transfer learning paradigm and leveraging the significant
industrial efforts in foundation model development [8, 11], the remaining
challenge lies in the fine-tuning process. The central issue is balancing the
adaptation of the initial model to sufficiently specialize it for the medical
task while avoiding overfitting to the training set, which would result in
poor generalization to the testing set. This balance has been explored through
various fine-tuning algorithms, such as regularized fine-tuning [64, 23].
However, parameter-efficient fine-tuning (PEFT) has recently gained traction
[63, 51, 32, 27], initially within the NLP community and subsequently in
computer vision. The concept involves either carefully selecting a subset of
parameters to fine-tune while keeping the rest frozen or introducing a small
number of new parameters during the fine-tuning step and freezing all
transferred parameters. While numerous PEFT architectures have emerged in NLP
and vision literature, a comprehensive empirical evaluation comparing their
effectiveness on a common benchmark is lacking in medical image analysis.
Moreover, we observe that PEFT architectures are not widely adopted in medical
vision tasks, despite the data-efficiency needs of the medical domain.

In this paper, we present an evaluation of state-of-the-art PEFT methods using
medical image analysis benchmark tasks. We compare 16 state-of-the-art PEFT
techniques across six medical datasets, encompassing both CNN and transformer
architectures, discriminative diagnosis tasks, and a novel –first-of-a-kind
demonstration of PEFT’s effectiveness– in a generative medical image synthesis
task. We experiment with architectures that match the size of recent
foundation models introduced for computer vision and medical image analysis
[39, 11]. In addition, we investigate aspects such as the trade-off between
PEFT effectiveness and data volume for the task at hand and the balance
between parameter count and efficacy. Our extensive experiments offer the
community of medical image analysis and medical vision valuable insights into
which PEFT methods are best suited for their task and, more broadly, establish
the first comprehensive comparison benchmark for PEFT in medical vision.

Our contributions can be summarised by the following questions and their
answers:

Q1: How effective is PEFT for low data scenarios? A1: Given a large pre-
trained model, benefits from PEFT increase as data volume decreases and model
size increases (Sec. 4.1).

Q2: Can PEFT improve transfer to discriminative medical tasks? A2: Yes, three
methods achieve consistent gains compared to full fine-tuning, two of which
also significantly reduce the computational cost of tuning (Sec. 4.2).

Q3: Can PEFT improve costly text-to-image generation? A3: PEFT methods are
less reliable in this setting but can still offer performance gains, with
moderate reductions to computational cost (Sec. 4.3).

PEFT Method | Paper | Summary | CNNs | ViTs | PEFT Type
---|---|---|---|---|---
| Task-Specific Adapters
---
(TSA)
Li _et al_. [42] | Cross-domain few-shot learning by inserting learnable modules. | ✓ | ✗ | Additive
BatchNorm Tuning | Frankle _et al_. [22] |  | Training only BatchNorm layers (even with random initialization)
---
leads to high performance in CNNs.
✓ | ✗ | Selective
Bias Tuning | Cai _et al_. [10] |  | Propose TinyTL framework that learns only bias modules
---
for parameter-efficient on-device learning.
✓ | ✗ | Selective
| Scale-Shift Features
---
(SSF)
Lian _et al_. [44] |  | Adapt a pre-trained model to downstream datasets by
---
introducing parameters that modulate the extracted features.
✓ | ✓ | Additive
Attention Tuning | Touvron _et al_. [58] |  | Fine-tuning attention layers is sufficient to adapt ViTs to
---
different classification tasks.
✗ | ✓ | Selective
LayerNorm Tuning | Basu _et al_. [6] |  | Fine-tuning LayerNorm parameters is a strong baseline for
---
few-shot adaptation.
✗ | ✓ | Selective
BitFit | Zaken _et al_. [7] |  | Fine-tuning the bias terms in a transformer is competitive
---
or better than full-fine-tuning.
✗ | ✓ | Selective
LoRA | Hu _et al_. [32] |  | Training injected rank decomposition matrices in transformers
---
is on-par or better than full-fine-tuning.
✗ | ✓ | Additive
AdaptFormer | Chen _et al_. [13] |  | Adding lightweight modules increases a ViT’s transferability for
---
different image and video tasks.
✗ | ✓ | Additive
SV-Diff | Han _et al_. [26] |  | Fine-tuning singular values of weight matrices is a parameter-efficient
---
adapter for text-to-image generation models.
| U-Net and Text-
---
Encoder in SD
Additive

Table 1: Summary of the Parameter-Efficient Fine-Tuning (PEFT) methods
included in this evaluation, highlighting the specific model type they are
designed for and their respective categories.

##  2 Related Work

###  2.1 Adapters

Additive Methods. Adapters are new modules added among the layers of a pre-
trained network to reconfigure a given pre-trained model for a downstream
task. The idea was originally introduced for learning representations that
could aid in the analysis of multiple, diverse domains. [50, 51]. Li _et al_.
[42] introduced Task-Specific Adapters (TSA) for learning multiple domains but
with limited examples (cross-domain few-shot learning). Houlsby _et al_. [31]
extended [50] for natural language processing by applying adapters to
transformers. Hu et al. [32] build upon [1] and introduce low-rank matrices
into the self-attention layer of transformer-based models. This approach,
termed LoRA, introduces no additional inference latency, as seen in [31], and
has demonstrated superior performance over traditional model fine-tuning
across diverse datasets and tasks. Lian _et al_. [44] propose a different
parameter-efficient fine-tuning approach through Scaling and Shifting the
Features (SSF) extracted by a pre-trained model. The difference in upstream
and downstream data distribution poses a challenge to adapting pre-trained
models to downstream datasets. SSF introduces parameters, after each network
operation, that modulate (scale and shift) the extracted features of the
downstream dataset such that they fall in a discriminative space. Chen _et
al_. [13] introduced a novel PEFT approach designed for vision transformers
for image and video recognition (AdaptFormer) and demonstrated that adding
only 2% extra parameters to a vision transformer can outperform fully-
finetuned networks on multiple benchmarks.

Selective Methods. This line of approach does not insert any new modules but
finetunes a subset of the original pre-trained model. BitFit [7] proposes
finetuning only the bias terms in a transformer-based model. In addition to
being parameter-efficient, BitFit also alleviates catastrophic forgetting and
matches full fine-tuning performance. Cai _et al_. [10] have drawn a similar
parallel by fine-tuning only the bias terms in CNNs for on-device learning.
Similarly, Frankle _et al_. [22] reveal the expressivity of parameters in the
batch normalization layers, particularly, in shifting and rescaling random
features. The SSF approach [44] described earlier, was developed on this
understanding. Similar properties have also been revealed in Vision
transformers [18]. Touvron _et al_. [58] show that fine-tuning only attention
layers in vanilla ViTs is an effective strategy. Basu _et al_. [6]
demonstrated that fine-tuning the parameters of normalization layers is very
effective for few-shot classification.

Prompt Tuning. The concept of appending prompts to large language models has
enabled powerful zero-shot and few-shot performance [9, 56]. Prompt tuning
[40] introduced the idea of appending the input in language models with a
trainable tensor, termed soft prompt. Jia _et al_. [34] extended this concept
for vision transformers by proposing two schemes, VPT-Deep and VPT-Shallow.
Similar to VPT-Deep, Li _et al_. [43] advocated the idea of appending
trainable parameters in all hidden layers . Qin _et al_. [49] proposed
adopting an autoencoder for decompressing the soft prompts to improve the
convergence in [40]. A summary of different PEFT methods along with their
categorization is given in Table 1.

###  2.2 PEFT for Text-to-Image Generation

Diffusion models [29] have led to state-of-the-art results in a variety of
tasks such as text-to-image generation [53, 5, 55], image synthesis [17],
density estimation [38] and many others. Recently, PEFT methods for text-to-
image diffusion models have been proposed. Xie _et al_. [63] proposed DiffFit,
for diffusion models based on transformers [48]. They found that fine-tuning
the bias terms is a strong baseline for downstream fine-tuning and further
introduced learnable scaling factors at specific positions of the model for
efficient adaptation. Moon _et al_. [46] identified fine-tuning the attention
modules as another effective strategy and introduced an adapter that improves
image-generation quality. Xiang _et al_. [62] presented the procedure for
parameter-efficient fine-tuning in diffusion models by studying the design
space of adapters in terms of position and function form. Han _et al_. [26]
propose a new method, termed “SV-Diff”, for efficient personalization of text-
to-image diffusion models. They designed a compact parameter space by fine-
tuning singular values of weight matrices. This method is more parameter-
efficient than existing approaches such as LoRA [32].

###  2.3 PEFT for Medical Image Analysis

There has been limited adoption of PEFT techniques within medical image
analysis. Fischer _et al_. [21] modify a vanilla U-Net [54] with class-
dependent learnable prompt tokens for semantic segmentation. Their approach
almost matches the performance of a fully finetuned model on two medical
datasets. Rodríguez _et al_. [57] study adapters designed for dense prediction
tasks to improve medical image segmentation. Experiments with the recently
proposed Segment Anything Model (SAM) [39] have revealed its inefficacy in the
medical image analysis domain [16]. To overcome this, Wu _et al_. [61] combine
the SAM model with adapters [31] and outperform existing state-of-the-art
methods on several downstream medical segmentation tasks. Similarly, Zhang _et
al_. [66] combined SAM with LoRA. In this work, we perform the first wide
benchmarking study that applies PEFT techniques to diverse tasks in the
medical image analysis domain, using state-of-the-art architectures.

##  3 Backgrund

###  3.1 Problem Definition

The general problem definition for fine-tuning can be formalized as follows.
Let $f$ be a pre-trained model parameterized by $\theta$, $\ell$ be a loss
function we wish to minimize and $\mathcal{D}=\\{(x_{i},y_{i})\\}_{i}^{N}$ be
the downstream dataset of interest, consisting of inputs $x_{i}$ and their
targets $y_{i}$. Starting from the initialization $\theta=\theta_{{}_{0}}$,
where $\theta_{{}_{0}}$ are the weights from pre-training, our objective is
then to optimize by gradient descent the total loss

$L=\frac{1}{N}\sum_{i=1}^{N}\ell(f(x_{i};\theta),y_{i}).$ (1)

Due to resource constraints, full fine-tuning like this is not always
possible. It can also be suboptimal to tune the entirety of network weights,
as many layers may have learned generally applicable features. Parameter-
Efficient Fine-Tuning provides options in these cases, which fall into two
broad families. Selective methods rely on optimising only a subset of model
parameters, $\phi\in\theta$. This could be a subset of the layers or a
specific type of parameter like batch norm. Additive methods instead introduce
new parameters such that the full set becomes
$\theta^{\prime}=\\{\theta,\phi\\}$ where $\phi$ can be as simple as a new
classifier layer or carefully designed adapters. For both families of methods,
the update rule becomes

$\phi=\phi-\eta\nabla_{\phi}L,$ (2)

where $\eta$ is the learning rate.

###  3.2 PEFT Methods For Comparison

In this section, we formally define the different fine-tuning protocols used
in the analysis. We begin with a downstream dataset $D$ and a feature
extractor $f_{\theta}$ (pre-trained CNN (ResNet50) or a ViT (Base/Large/Huge))
expected to produce generalizable representations for diverse tasks. First, we
freeze all the weights of this feature extractor and enable either an existing
subset or a newly added parameter set according to the fine-tuning protocol.

In selective tuning methods, we permit specific parameters to be trainable
based on the selected algorithm. For instance, for protocols like BatchNorm
and Bias Tuning, the parameters of the ‘BatchNorm2d’ layers or the ‘bias’
terms are respectively made trainable. The training protocols for other
selective methods are given in Appendix (Sec 6.3).

In TSA, our objective is to learn task-specific weights $\phi$ to obtain the
task-adapted classifier $f_{(\theta,\phi)}$. Next, we minimize the cross-
entropy loss $L$ over the samples in the downstream dataset $D$ w.r.t the
task-specific weights $\phi$. Li _et al_. [42] recommend the parallel adapter
configuration. The output of the l-th layer of the feature extractor
$f_{\theta}$ can be combined with the task-specific adapters $r_{\phi}$ for an
input tensor $h\in\mathbb{R}^{W\times H\times C}$ in a parallel configuration
using Equation 3,

$f_{(\theta_{l},\phi)}(h)=r_{\phi}(h)+f_{\theta_{l}}(h).$ (3)

In the SSF method, feature modulation is achieved by introducing scale
($\gamma$) and shift ($\beta$) parameters following each operation in the
model. The previous operation’s output is multiplied by the scale parameter
through a dot product and combined with the shift factor. Therefore, for a
given input $x$, the output $y$ is calculated using the formula $y=\gamma\cdot
x+\beta$.

An AdaptFormer module (AdaptMLP) consists of two branches wherein the first
branch is identical to the MLP block of a vanilla transformer while the second
branch consists of a down-projection ($W_{down}$), a ReLU layer, an up-
projection ($W_{up}$), and a scaling factor ($s$). The adapted features
($x_{adap}$), obtained using Equation 4, are combined with the original
features entering the AdaptMLP block ($x_{orig}$) through a residual
connection, as defined in Equation 5.

$\displaystyle x_{adap}=ReLU(LN(x_{orig})\cdot W_{down})\cdot W_{up}$ (4)
$\displaystyle x_{final}=MLP(LN(x_{orig}))+s.x_{adap}+x_{orig}$ (5)

LoRA is based on the concept that, during adaptation, weight updates exhibit
low intrinsic rank. Consequently, when a pre-trained weight matrix $W_{0}$ is
updated, the change ($\Delta W$) is characterized by a low-rank decomposition
operation with rank $r$, as shown in Eqn. 6 where $B\in\mathbb{R}^{d\times r}$
and $A\in\mathbb{R}^{r\times k}$,

$W_{0}+\Delta W=W_{0}+BA.$ (6)

SV-Diff performs Singular Value Decomposition (SVD) of the weight matrices of
a pre-trained diffusion model (Equation 7) and optimizes the spectral shift
($\delta$), defined as the difference between singular values and of the
updated and original weight matrix. The update rule is defined in Eqn. 8,

$\displaystyle W=U\Sigma
V^{\intercal}\quad\text{with}\quad\Sigma=\text{diag}(\sigma),$ (7)
$\displaystyle
W_{\delta}=U\Sigma_{\delta}V^{\intercal}\quad\text{with}\quad\Sigma_{\delta}=\text{diag}(\text{ReLU}(\sigma+\delta)).$
(8)

Figure 1: Plots showing the performance comparison for Full Fine-tuning,
BitFit and LoRA with varying downstream dataset size for ViT-Base and ViT-
Large models. Figure 2: Figure showing text-to-image generation examples with
the ground truth in the ascending average rank order (best five) for two data
regimes. The input prompt for the generated samples is: “No acute
cardiopulmonary process.”

##  4 Experiments

###  4.1 How Effective is PEFT For Low Data Scenarios?

Setup. In order to investigate the effectiveness of various fine-tuning
approaches in relation to the downstream data volume, we utilized the HAM10000
dataset [59] and employed three distinct fine-tuning methods, namely Full
Fine-tuning, BitFit, and LoRA, in combination with two different encoders, ViT
Base and ViT Large. The performance of each method was evaluated by measuring
the F1-Score at various dataset sizes, commencing with the entire sample size
of 7,511 images (100%) and progressively reducing it to a minimum of 75 images
(1%). To account for potential variability in the results, we report the
average performance across three random seeds.

Results. The results are shown in Figure 1. Starting with ViT Base on the
left, we find that when using 100% of available downstream data, full fine-
tuning is optimal, closely followed by LoRA. As the availability decreases,
however, the benefits from PEFT approaches increase. The crossover is at 50%,
when all approaches are approximately equal. For smaller data sizes, both PEFT
approaches consistently outperform full FT, with LoRA providing gains of up to
6% over the baseline. For ViT Large, the trend is similar, but the crossover
now differs between the PEFT approaches. LoRA overtakes the baseline as early
as 80% while BitFit is only better at data volumes below 20%. Our take-home
message here is that when data is scarce, coupled with a large upstream model,
it becomes especially important to consider parameter-efficient tuning.

###  4.2 Can PEFT Improve Transfer to Discriminative Medical Tasks?

Setup. In our discriminative experiments, we use five diverse datasets widely
recognized in the medical image analysis community for image classification
tasks, BreastUS [2], HAM10000 [59], Fitzpatrick17K [25, 24], Standardized
Multi-Channel Dataset for Glaucoma (SMDG) [36, 37], and RSNA Pneumonia
Detection Dataset [47]. The experiments employ ResNet50 [28] and ViT
(Base/Large/Huge) [18] as encoders. All CNN experiments employed ResNet50 pre-
trained on ImageNet [15] while all ViT variants were pre-trained on
ImageNet-21k [52]. Hyper-Parameter Optimization (HPO) was utilized to
determine the optimal set of hyperparameters for each experiment. To ensure
the timely completion of our search, the ASHA hyperparameter optimization
algorithm [41] was adopted. For each dataset, various fine-tuning methods were
applied using the hyperparameters obtained from the search, and performance
was compared using a separate test set. During training, early stopping was
implemented, halting the training process if the performance on the validation
set did not improve for a predefined number of epochs.

Dataset Method | Full FT |  | Linear
---
Probing
TSA |  | BN
---
Tuning
| Bias
---
Tuning
SSF
| (23.5M) | (3.8-7.2K) | (10.6M) | (59.1K) | (32.7K) | (60.6K)
BreastUS (584) | $0.72\pm 1.1$ | $0.61\pm 1.3$ | $0.90\pm 0.8$ | $0.92\pm 0.9$ | $0.89\pm 1.2$ | $\textbf{0.94}\pm\textbf{0.7}$
FitzPatrick (5809) | $\textbf{0.71}\pm\textbf{0.4}$ | $0.66\pm 0.8$ | $0.69\pm 1.4$ | $0.67\pm 1.1$ | $0.64\pm 1.3$ | $\textbf{0.71}\pm\textbf{0.7}$
HAM10000 (7511) | $0.87\pm 1.2$ | $0.82\pm 0.6$ | $0.86\pm 1.0$ | $0.84\pm 0.6$ | $0.70\pm 1.0$ | $\textbf{0.89}\pm\textbf{0.9}$
SMDG (9852) | $0.75\pm 0.9$ | $0.69\pm 1.0$ | $\textbf{0.85}\pm\textbf{0.7}$ | $0.83\pm 1.4$ | $0.73\pm 0.6$ | $0.84\pm 0.9$
Pneumonia (20412) | $0.86\pm 1.4$ | $0.80\pm 0.4$ | $0.86\pm 1.1$ | $0.84\pm 1.5$ | $0.85\pm 1.9$ | $\textbf{0.87}\pm\textbf{1.2}$
Average F1 Score | 0.77 | 0.72 | 0.83 | 0.82 | 0.76 | 0.85
Average Rank | 2.8 | 5.2 | 2.2 | 3.2 | 4.6 | 1.2
Table 2: Comparing different fine-tuning methods for ImageNet pre-trained
ResNet50. Dataset size and parameter count are indicated in brackets. The best
result for each dataset is highlighted, and the average rank for each fine-
tuning method is shown at the end.

Figure 3: Performance v/s Parameter Count for ResNet50 and ViT-Base Encoders.
The marker size indicates the tunable parameter count for each method.

Results. We present the results for ResNet-50 in Table 2. Given its
convolutional architecture, ResNet-50 is compatible with certain PEFT methods
but not others. Overall, full fine-tuning tends to outperform basic linear
probing. Observations from the BreastUS and SMDG datasets indicate that most
PEFT methods enhance performance beyond the full FT baseline. The SSF method,
despite only tuning 60K parameters (0.25%), improves performance by up to 22%.
While gains on HAM10000, FitzPatrick and Pneumonia are more modest, the
previous section has discussed how these results could potentially vary with
changes in data volume and model size. Overall, SSF emerges as the top-
performing method based on average F1 score and ranking. Full fine-tuning and
TSA present a close tie with the latter emerging on top. BatchNorm and bias
tuning perform better than linear probing which turns out to be the worst
strategy. Overall, the greatest gains are observed in the smallest dataset
(BreastUS), however, the performance gap between full fine-tuning and PEFT
methods minimizes with an increase in dataset size. For Transformer models in
Tab. 3, the situation is similar. The biggest gains over full FT are on
BreastUS and SMDG, while linear probing underperforms here as well. The top
PEFT method is LoRA, for both average F1 score and rank, across all five
datasets. AdaptFormer does not perform well and even falls behind linear
probing for ViT Large. This can be attributed to the fact that this method was
mainly designed for video recognition tasks. We also see that the benefits of
PEFT increase slightly as the model size increases, with a 4% improvement for
ViT Base going to 6% for ViT Huge. This is an interesting finding, and agrees
with Sec. 4.1, as the proportion of parameters tuned actually decreases for
the larger models.

Figure 3 illustrates the trade-off between each method’s performance and
parameter count. This comparison is crucial as different applications may
prioritize either superior performance or computational efficiency. For the
results produced by the ResNet50 (shown on the left), each PEFT method lies on
the Pareto frontier, indicating that a specific method could be selected based
on the prioritization of either performance or cost. Remarkably, the SSF
method stands out by delivering high performance at a significantly reduced
cost. In the case of the ViT-B model, LoRA emerges as the prominent choice,
outpacing SSF while maintaining a similar computational expense.

To answer our question _can PEFT Improve Transfer to Discriminative Medical
Tasks?_ Yes, TSA, SSF and LoRA provide consistent improvements over full fine-
tuning, and there is room to prefer less resource-intensive methods if
necessary, requiring as little as 0.25% of parameters.

Encoder |  Dataset Method | Full FT |  | Linear
---
Probing
| Attention
---
Tuning
BitFiT | LoRA | SSF | Adaptformer |  | LayerNorm
---
Tuning
ViT Base | BreastUS (584) | $0.82\pm 1.2$ | $0.79\pm 0.7$ | $0.93\pm 1.4$ | $\textbf{0.97}\pm\textbf{1.3}$ | $0.94\pm 0.6$ | $0.95\pm 0.9$ | $0.95\pm 0.7$ | $0.88\pm 1.1$
FitzPatrick (5,809) | $0.80\pm 1.3$ | $0.74\pm 0.6$ | $0.76\pm 1.3$ | $0.71\pm 1.6$ | $\textbf{0.82}\pm\textbf{1.4}$ | $0.77\pm 0.7$ | $0.72\pm 1.1$ | $0.73\pm 1.2$
HAM10000 (7,511) | $\textbf{0.91}\pm\textbf{1.4}$ | $0.72\pm 0.5$ | $0.86\pm 1.2$ | $0.87\pm 1.8$ | $\textbf{0.91}\pm\textbf{1.3}$ | $0.88\pm 0.8$ | $0.76\pm 1.2$ | $0.85\pm 1.3$
SMDG (9,852) | $0.80\pm 1.6$ | $0.60\pm 0.6$ | $0.84\pm 1.8$ | $0.66\pm 1.4$ | $\textbf{0.86}\pm\textbf{1.5}$ | $0.85\pm 0.9$ | $0.60\pm 1.3$ | $0.80\pm 1.4$
Pneumonia (20,412) | $0.87\pm 1.7$ | $0.86\pm 0.4$ | $0.85\pm 1.1$ | $0.87\pm 1.2$ | $0.86\pm 0.8$ | $\textbf{0.88}\pm\textbf{1.0}$ | $0.83\pm 0.9$ | $0.87\pm 1.7$
| Average F1 Score | 0.84 | 0.74 | 0.85 | 0.82 | 0.88 | 0.87 | 0.77 | 0.83
ViT Large | BreastUS (584) | $0.84\pm 1.8$ | $0.73\pm 0.7$ | $0.86\pm 1.3$ | $\textbf{0.95}\pm\textbf{1.4}$ | $0.93\pm 1.3$ | $0.92\pm 1.8$ | $\textbf{0.95}\pm\textbf{1.1}$ | $0.88\pm 1.4$
| FitzPatrick (5,809) | $\textbf{0.82}\pm\textbf{1.4}$ | $0.74\pm 0.5$ | $0.77\pm 1.2$ | $0.74\pm 1.5$ | $\textbf{0.82}\pm\textbf{1.9}$ | $0.80\pm 1.3$ | $0.72\pm 1.2$ | $0.78\pm 1.3$
| HAM10000 (7,511) | $\textbf{0.90}\pm\textbf{1.6}$ | $0.82\pm 0.8$ | $0.88\pm 1.4$ | $0.86\pm 1.1$ | $0.89\pm 1.5$ | $0.88\pm 1.7$ | $0.74\pm 1.0$ | $0.87\pm 1.7$
| SMDG (9,852) | $0.81\pm 1.5$ | $0.77\pm 0.6$ | $0.84\pm 1.5$ | $0.83\pm 1.9$ | $0.83\pm 1.2$ | $\textbf{0.87}\pm\textbf{1.2}$ | $0.63\pm 1.3$ | $0.85\pm 1.5$
| Pneumonia (20,412) | $0.80\pm 1.8$ | $0.78\pm 0.9$ | $0.81\pm 1.5$ | $0.80\pm 1.4$ | $\textbf{0.82}\pm\textbf{1.1}$ | $0.80\pm 1.0$ | $0.78\pm 1.4$ | $0.80\pm 1.6$
| Average F1 Score | 0.83 | 0.77 | 0.83 | 0.84 | 0.86 | 0.85 | 0.76 | 0.84
ViT Huge | BreastUS (584) | $0.92\pm 1.8$ | $0.67\pm 0.9$ | $0.89\pm 1.5$ | $\textbf{0.96}\pm\textbf{1.2}$ | $0.86\pm 1.8$ | $\textbf{0.96}\pm\textbf{1.1}$ | $0.93\pm 1.0$ | $0.92\pm 1.4$
| FitzPatrick (5,809) | $0.69\pm 1.3$ | $0.72\pm 0.6$ | $0.70\pm 1.3$ | $0.72\pm 1.2$ | $\textbf{0.78}\pm\textbf{1.5}$ | $0.73\pm 1.1$ | $0.72\pm 1.4$ | $0.72\pm 0.8$
| HAM10000 (7,511) | $0.74\pm 1.7$ | $0.74\pm 0.7$ | $0.77\pm 1.5$ | $0.71\pm 1.4$ | $0.87\pm 1.1$ | $0.70\pm 0.7$ | $0.73\pm 1.0$ | $0.72\pm 1.7$
| SMDG (9,852) | $0.73\pm 1.5$ | $0.64\pm 1.1$ | $0.72\pm 1.4$ | $0.64\pm 0.9$ | $\textbf{0.83}\pm 1.7$ | $0.67\pm 1.1$ | $0.64\pm 1.2$ | $0.67\pm 1.3$
| Pneumonia (20,412) | $0.78\pm 1.6$ | $0.76\pm 1.3$ | $0.78\pm 0.9$ | $0.79\pm 1.5$ | $\textbf{0.81}\pm\textbf{1.7}$ | $0.79\pm 1.1$ | $0.78\pm 1.1$ | $0.78\pm 1.2$
| Average F1 Score | 0.77 | 0.71 | 0.77 | 0.76 | 0.83 | 0.77 | 0.76 | 0.76
Combined Average Rank |  | 4.1 | 6.7 | 4.5 | 4.5 | 2.4 | 3.1 | 6.0 | 4.7

Table 3: Results with different ViT encoders (base/ large/ huge). Dataset size
and parameter count are indicated in brackets. The best result for each
dataset is highlighted, and the average rank for each fine-tuning method is
shown at the end.

###  4.3 Can PEFT Improve Costly Text-to-Image Generation?

Setup. We use the MIMIC-CXR dataset (version 2.0.0) [35]. Following the
recommendations in [12], we begin with fine-tuning the U-Net component
(keeping text-encoder and VAE frozen) of the stable diffusion pipeline for
different sizes of the downstream dataset (2262 and 226 samples, representing
10% and 1% of the original MIMIC-CXR dataset). We retained only those studies
where the associated text reports had a character length between 7 and 77
tokens. Finally, we created a separate training set (subgroups P10 - P18) and
a test set (subgroup P19). For analysis, we compare the full-finetuning of
U-Net with 7 different PEFT methods and report the FID Score over 1000 test
images averaged across three random seeds.

Results. Our results for this section can be found in (Table 4). Example
images generated using different fine-tuning methods for the two scenarios are
shown in Figure 2. For both data volumes, only a subset of PEFT methods yields
performance enhancements over full fine-tuning. Simply fine-tuning attention
layers turns out to be an effective strategy which can be further improved by
simultaneous fine-tuning of attention, bias, and normalization layers
(bias+norm+attention). This combination also outperforms full fine-tuning in
both data regimes. Interestingly, both these methods require more
computational resources than other PEFT methods but fine-tune merely 31% of
parameters relative to the full FT. Consequently, while current PEFT methods
may not be as dependable for medical text-to-image generation as they are for
classification tasks, they possess the potential to enhance performance and
simultaneously decrease computational expenditure.

PEFT |  | Full FT
---
(85.9M)
| Attention
---
(26.7M)
| Bias
---
(343K)
| Norm
---
(200K)
| Bias+Norm
---
(443K)
| Bias+Norm+Attention
---
(26.7M)
| LoRA (SD)
---
(797K)
| SV-Diff
---
(21.4K)
FID @ 2262 (10%) | $7.15\pm 1.1$ | $8.35\pm 1.3$ | $22.43\pm 0.9$ | $20.25\pm 1.2$ | $31.26\pm 1.1$ | $\textbf{6.95}\pm\textbf{0.8}$ | $11.84\pm 0.7$ | $14.80\pm 0.8$
FID @ 226 (1%) | $10.43\pm 0.9$ | $8.44\pm 0.7$ | $32.90\pm 1.3$ | $29.67\pm 1.2$ | $36.76\pm 0.7$ | $\textbf{8.28}\pm\textbf{0.9}$ | $14.88\pm 1.3$ | $16.48\pm 1.4$
Average FID | 8.79 | 8.39 | 27.66 | 24.96 | 34.01 | 7.61 | 13.18 | 15.64
Combined Average Rank | 2.5 | 2.5 | 7 | 6 | 8 | 1 | 5 | 5

Table 4: Comparing different strategies for fine-tuning the U-Net subcomponent
of the stable diffusion pipeline text-to-image generation of chest x-rays.
Fine-tuning Bias, Normalization, and Attention layers together
(Bias+Norm+Attention) result in better performance (lower FID score) across
both data volumes.

##  5 Conclusion

In this work, we have performed the first parameter-efficient fine-tuning
evaluation for the medical image analysis domain. It has covered a diverse set
of approaches and datasets and includes both discriminative and generative
tasks. Our findings are (i) PEFT methods become more dominant with decreasing
dataset size and increasing model size as full fine-tuning presents a risk of
overfitting in this scenario; (ii) for discriminative tasks, the benefits from
PEFT are especially prominent for low to medium-scale datasets, which are
particularly common in the medical domain. Moreover, PEFT greatly reduces the
computational and memory requirements, thereby making the application and
adoption of large models much more practical in clinical settings; (iii) for
text-to-image generation, the PEFT methods are less reliably effective,
suggesting more work is needed in this area. However, generation quality can
still be improved with a third of the parameters of full fine-tuning. As a
recommendation, SSF and LoRA could be employed while adapting large
convolution and transformer-based networks to medical tasks, respectively.
Overall, PEFT methods should be an integral part of a medical AI
practitioner’s toolbox.

## References

  * [1] Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta.  Intrinsic dimensionality explains the effectiveness of language model fine-tuning.  arXiv preprint arXiv:2012.13255, 2020.
  * [2] Walid Al-Dhabyani, Mohammed Gomaa, Hussien Khaled, and Aly Fahmy.  Dataset of breast ultrasound images.  Data in brief, 28:104863, 2020.
  * [3] Shekoofeh Azizi, Laura Culp, Jan Freyberg, Basil Mustafa, Sebastien Baur, Simon Kornblith, Ting Chen, Patricia MacWilliams, S. Sara Mahdavi, Ellery Wulczyn, Boris Babenko, Megan Wilson, Aaron Loh, Po-Hsuan Cameron Chen, Yuan Liu, Pinal Bavishi, Scott Mayer McKinney, Jim Winkens, Abhijit Guha Roy, Zach Beaver, Fiona Ryan, Justin Krogue, Mozziyar Etemadi, Umesh Telang, Yun Liu, Lily Peng, Greg S. Corrado, Dale R. Webster, David Fleet, Geoffrey Hinton, Neil Houlsby, Alan Karthikesalingam, Mohammad Norouzi, and Vivek Natarajan.  Robust and efficient medical imaging with self-supervision, 2022.
  * [4] Shekoofeh Azizi, Basil Mustafa, Fiona Ryan, Zachary Beaver, Jan Freyberg, Jonathan Deaton, Aaron Loh, Alan Karthikesalingam, Simon Kornblith, Ting Chen, Vivek Natarajan, and Mohammad Norouzi.  Big self-supervised models advance medical image classification.  In ICCV, 2021.
  * [5] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al.  ediffi: Text-to-image diffusion models with an ensemble of expert denoisers.  arXiv preprint arXiv:2211.01324, 2022.
  * [6] Samyadeep Basu, Daniela Massiceti, Shell Xu Hu, and Soheil Feizi.  Strong baselines for parameter efficient few-shot fine-tuning.  arXiv preprint arXiv:2304.01917, 2023.
  * [7] Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel.  BitFit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1–9, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * [8] Rishi Bommasani et al.  On the opportunities and risks of foundation models.  arXiv:2108.07258, 2021.
  * [9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [10] Han Cai, Chuang Gan, Ligeng Zhu, and Song Han.  Tinytl: Reduce memory, not parameters for efficient on-device learning.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 11285–11297. Curran Associates, Inc., 2020.
  * [11] Pierre Chambon, Christian Bluethgen, Jean-Benoit Delbrouck, Rogier Van der Sluijs, Małgorzata Połacin, Juan Manuel Zambrano Chaves, Tanishq Mathew Abraham, Shivanshu Purohit, Curtis P. Langlotz, and Akshay Chaudhari.  Roentgen: Vision-language foundation model for chest x-ray generation, 2022.
  * [12] Pierre Chambon, Christian Bluethgen, Curtis P Langlotz, and Akshay Chaudhari.  Adapting pretrained vision-language foundational models to medical imaging domains.  arXiv preprint arXiv:2210.04133, 2022.
  * [13] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo.  Adaptformer: Adapting vision transformers for scalable visual recognition.  arXiv preprint arXiv:2205.13535, 2022.
  * [14] Jeffrey De Fauw, Joseph R Ledsam, Bernardino Romera-Paredes, Stanislav Nikolov, Nenad Tomasev, Sam Blackwell, Harry Askham, Xavier Glorot, Brendan O’Donoghue, Daniel Visentin, et al.  Clinically applicable deep learning for diagnosis and referral in retinal disease.  Nature medicine, 24(9):1342–1350, 2018.
  * [15] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
  * [16] Ruining Deng, Can Cui, Quan Liu, Tianyuan Yao, Lucas W Remedios, Shunxing Bao, Bennett A Landman, Lee E Wheless, Lori A Coburn, Keith T Wilson, et al.  Segment anything model (sam) for digital pathology: Assess zero-shot segmentation on whole slide imaging.  arXiv preprint arXiv:2304.04155, 2023.
  * [17] Prafulla Dhariwal and Alexander Nichol.  Diffusion models beat gans on image synthesis.  Advances in Neural Information Processing Systems, 34:8780–8794, 2021.
  * [18] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, 2021.
  * [19] Raman Dutt, Dylan Mendonca, Huai Ming Phen, Samuel Broida, Marzyeh Ghassemi, Judy Gichoya, Imon Banerjee, Tim Yoon, and Hari Trivedi.  Automatic localization and brand detection of cervical spine hardware on radiographs using weakly supervised machine learning.  Radiology: Artificial Intelligence, 4(2):e210099, 2022.
  * [20] Andre Esteva, Brett Kuprel, Roberto A Novoa, Justin Ko, Susan M Swetter, Helen M Blau, and Sebastian Thrun.  Dermatologist-level classification of skin cancer with deep neural networks.  nature, 542(7639):115–118, 2017.
  * [21] Marc Fischer, Alexander Bartler, and Bin Yang.  Prompt tuning for parameter-efficient medical image segmentation.  arXiv preprint arXiv:2211.09233, 2022.
  * [22] Jonathan Frankle, David J. Schwab, and Ari S. Morcos.  Training batchnorm and only batchnorm: On the expressive power of random features in cnns.  CoRR, abs/2003.00152, 2020.
  * [23] Henry Gouk, Timothy Hospedales, et al.  Distance-based regularisation of deep networks for fine-tuning.  In International Conference on Learning Representations, 2020.
  * [24] Matthew Groh, Caleb Harris, Roxana Daneshjou, Omar Badri, and Arash Koochek.  Towards transparency in dermatology image datasets with skin tone annotations by experts, crowds, and an algorithm.  arXiv preprint arXiv:2207.02942, 2022.
  * [25] Matthew Groh, Caleb Harris, Luis Soenksen, Felix Lau, Rachel Han, Aerin Kim, Arash Koochek, and Omar Badri.  Evaluating deep neural networks trained on clinical images in dermatology with the fitzpatrick 17k dataset.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1820–1828, 2021.
  * [26] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang.  Svdiff: Compact parameter space for diffusion fine-tuning.  arXiv preprint arXiv:2303.11305, 2023.
  * [27] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig.  Towards a unified view of parameter-efficient transfer learning.  In International Conference on Learning Representations, 2022.
  * [28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [29] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  Advances in Neural Information Processing Systems, 33:6840–6851, 2020.
  * [30] Arthur S Hong, David Levin, Laurence Parker, Vijay M Rao, Dennis Ross-Degnan, and J Frank Wharam.  Trends in diagnostic imaging utilization among medicare and commercially insured adults from 2003 through 2016.  Radiology, 294(2):342–350, 2020.
  * [31] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly.  Parameter-efficient transfer learning for nlp.  In International Conference on Machine Learning, pages 2790–2799. PMLR, 2019.
  * [32] Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen.  LoRA: Low-rank adaptation of large language models.  In International Conference on Learning Representations, 2022.
  * [33] Shih-Cheng Huang, Anuj Pareek, Malte Jensen, Matthew P. Lungren, Serena Yeung, and Akshay S. Chaudhari.  Self-supervised learning for medical image classification: a systematic review and implementation guidelines.
  * [34] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim.  Visual prompt tuning.  In Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXIII, page 709–727, Berlin, Heidelberg, 2022. Springer-Verlag.
  * [35] Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng.  Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports.  Scientific data, 6(1):317, 2019.
  * [36] Riley Kiefer.  Smdg, a standardized fundus glaucoma dataset, 2023.
  * [37] Riley Kiefer, Jessica Steen, Muhammad Abid, Mahsa R Ardali, and Ehsan Amjadian.  A survey of glaucoma detection algorithms using fundus and oct images.  In 2022 IEEE 13th Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON), pages 0191–0196. IEEE, 2022.
  * [38] Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho.  Variational diffusion models.  Advances in neural information processing systems, 34:21696–21707, 2021.
  * [39] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al.  Segment anything.  arXiv preprint arXiv:2304.02643, 2023.
  * [40] Brian Lester, Rami Al-Rfou, and Noah Constant.  The power of scale for parameter-efficient prompt tuning.  arXiv preprint arXiv:2104.08691, 2021.
  * [41] Lisha Li, Kevin Jamieson, Afshin Rostamizadeh, Katya Gonina, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar.  Massively parallel hyperparameter tuning, 2018.
  * [42] Wei-Hong Li, Xialei Liu, and Hakan Bilen.  Cross-domain few-shot learning with task-specific adapters.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7161–7170, 2022.
  * [43] Xiang Lisa Li and Percy Liang.  Prefix-tuning: Optimizing continuous prompts for generation.  arXiv preprint arXiv:2101.00190, 2021.
  * [44] Dongze Lian, Daquan Zhou, Jiashi Feng, and Xinchao Wang.  Scaling & shifting your features: A new baseline for efficient model tuning.  arXiv preprint arXiv:2210.08823, 2022.
  * [45] Scott Mayer McKinney, Marcin Sieniek, Varun Godbole, Jonathan Godwin, Natasha Antropova, Hutan Ashrafian, Trevor Back, Mary Chesus, Greg S Corrado, Ara Darzi, et al.  International evaluation of an ai system for breast cancer screening.  Nature, 577(7788):89–94, 2020.
  * [46] Taehong Moon, Moonseok Choi, Gayoung Lee, Jung-Woo Ha, and Juho Lee.  Fine-tuning diffusion models with limited data.  In NeurIPS 2022 Workshop on Score-Based Methods.
  * [47] Radiological Society of North America.  Rsna pneumonia detection dataset.  Available at: https://www.kaggle.com/competitions/rsna-pneumonia-detection-challenge/data, 2018\.  Accessed: October, 2022.
  * [48] William Peebles and Saining Xie.  Scalable diffusion models with transformers.  arXiv preprint arXiv:2212.09748, 2022.
  * [49] Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Jing Yi, Weize Chen, Zhiyuan Liu, Juanzi Li, Lei Hou, Peng Li, Maosong Sun, and Jie Zhou.  Exploring universal intrinsic task subspace via prompt tuning.  2021\.
  * [50] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi.  Learning multiple visual domains with residual adapters.  Advances in neural information processing systems, 30, 2017.
  * [51] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi.  Efficient parametrization of multi-domain deep neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8119–8127, 2018.
  * [52] Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik.  Imagenet-21k pretraining for the masses.  In J. Vanschoren and S. Yeung, editors, Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, volume 1. Curran, 2021.
  * [53] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684–10695, 2022.
  * [54] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi, editors, Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, pages 234–241, Cham, 2015. Springer International Publishing.
  * [55] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al.  Photorealistic text-to-image diffusion models with deep language understanding.  Advances in Neural Information Processing Systems, 35:36479–36494, 2022.
  * [56] Timo Schick and Hinrich Schütze.  It’s not just size that matters: Small language models are also few-shot learners.  arXiv preprint arXiv:2009.07118, 2020.
  * [57] Julio Silva-Rodríguez, Jose Dolz, and Ismail Ben Ayed.  Transductive few-shot adapters for medical image segmentation.  arXiv preprint arXiv:2303.17051, 2023.
  * [58] Hugo Touvron, Matthieu Cord, Alaaeldin El-Nouby, Jakob Verbeek, and Hervé Jégou.  Three things everyone should know about vision transformers.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXIV, pages 497–515. Springer, 2022.
  * [59] Philipp Tschandl, Cliff Rosendahl, and Harald Kittler.  The ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions.  Scientific data, 5(1):1–9, 2018.
  * [60] Martin J Willemink, Wojciech A Koszek, Cailin Hardell, Jie Wu, Dominik Fleischmann, Hugh Harvey, Les R Folio, Ronald M Summers, Daniel L Rubin, and Matthew P Lungren.  Preparing medical imaging data for machine learning.  Radiology, 295(1):4–15, 2020.
  * [61] Junde Wu, Rao Fu, Huihui Fang, Yuanpei Liu, Zhaowei Wang, Yanwu Xu, Yueming Jin, and Tal Arbel.  Medical sam adapter: Adapting segment anything model for medical image segmentation.  arXiv preprint arXiv:2304.12620, 2023.
  * [62] Chendong Xiang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu.  A closer look at parameter-efficient tuning in diffusion models.  arXiv preprint arXiv:2303.18181, 2023.
  * [63] Enze Xie, Lewei Yao, Han Shi, Zhili Liu, Daquan Zhou, Zhaoqiang Liu, Jiawei Li, and Zhenguo Li.  Difffit: Unlocking transferability of large diffusion models via simple parameter-efficient fine-tuning.  arXiv preprint arXiv:2304.06648, 2023.
  * [64] LI Xuhong, Yves Grandvalet, and Franck Davoine.  Explicit inductive bias for transfer learning with convolutional networks.  In International Conference on Machine Learning, pages 2825–2834. PMLR, 2018.
  * [65] Fabian SL Yii, Raman Dutt, Tom MacGillivray, Baljean Dhillon, Miguel Bernabeu, and Niall Strang.  Rethinking retinal image quality: Treating quality threshold as a tunable hyperparameter.  In Ophthalmic Medical Image Analysis: 9th International Workshop, OMIA 2022, Held in Conjunction with MICCAI 2022, Singapore, Singapore, September 22, 2022, Proceedings, pages 73–83. Springer, 2022.
  * [66] Kaidong Zhang and Dong Liu.  Customized segment anything model for medical image segmentation.  arXiv preprint arXiv:2304.13785, 2023.
  * [67] S Kevin Zhou, Hayit Greenspan, Christos Davatzikos, James S Duncan, Bram Van Ginneken, Anant Madabhushi, Jerry L Prince, Daniel Rueckert, and Ronald M Summers.  A review of deep learning in medical imaging: Imaging traits, technology trends, case studies with progress highlights, and future promises.  Proceedings of the IEEE, 109(5):820–838, 2021.


# Numerical Bifurcation Analysis of Conformal
Formulations of the Einstein Constraints

M. Holst<EMAIL_ADDRESS>and  V. Kungurtsev<EMAIL_ADDRESS>Department of Mathematics
University of California San Diego
La Jolla CA 92093

###### Abstract.

The Einstein constraint equations have been the subject of study for more than
fifty years. The introduction of the conformal method in the 1970’s as a
parameterization of initial data for the Einstein equations led to increased
interest in the development of a complete solution theory for the constraints,
with the theory for constant mean curvature (CMC) spatial slices and closed
manifolds completely developed by 1995. The first general non-CMC existence
result was establish by Holst et al. in 2008, with extensions to rough data by
Holst et al. in 2009, and to vacuum spacetimes by Maxwell in 2009. The non-CMC
theory remains mostly open; moreover, recent work of Maxwell on specific
symmetry models sheds light on fundamental non-uniqueness problems with the
conformal method as a parameterization in non-CMC settings. In parallel with
these mathematical developments, computational physicists have uncovered
surprising behavior in numerical solutions to the extended conformal thin
sandwich formulation of the Einstein constraints. In particular, numerical
evidence suggests the existence of multiple solutions with a quadratic fold,
and a recent analysis of a simplified model supports this conclusion. In this
article, we examine this apparent bifurcation phenomena in a methodical way,
using modern techniques in bifurcation theory and in numerical homotopy
methods. We first review the evidence for the presence of bifurcation in the
Hamiltonian constraint in the time-symmetric case. We give a brief
introduction to the mathematical framework for analyzing bifurcation
phenomena, and then develop the main ideas behind the construction of
numerical homotopy, or path-following, methods in the analysis of bifurcation
phenomena. We then apply the continuation software package AUTO to this
problem, and verify the presence of the fold with homotopy-based numerical
methods. We discuss these results and their physical significance, which lead
to some interesting remaining questions to investigate further.

###### Key words and phrases:

bifurcation theory, nonlinear PDE, non-uniqueness, solution folds, pitchfork
bifurcation, compact operators, Fredholm operators, Fredholm index, homotopy
methods, continuation methods, pseudo-arclength continuation, Einstein
constraint equations

PACS Numbers: 04.20.Ex, 02.30.Jr, 02.30.Sa, 04.25D-

MH was supported in part by NSF Awards 0715146 and 0915220, and by DOD/DTRA
Award HDTRA-09-1-0036. VK was supported in part by NSF Awards 0715146 and
0915220.

###### Contents

  1. 1 Introduction
  2. 2 Conformal Thin Sandwich Decomposition
  3. 3 Nonlinear Operators on Banach Spaces
  4. 4 Bifurcation Theory for Nonlinear Operators Equations
  5. 5 Numerical Bifurcation Theory
  6. 6 Setup for Hamiltonian Constraint Bifurcation
  7. 7 Numerical Results
  8. 8 Discussion
  9. 9 Conclusion
  10. 10 Acknowledgments

##  1\. Introduction

Einstein’s gravitational field equations for relating the space curvature at a
time slice to the stress-energy can be split into a set of evolution and
constraint equations. The four constraint equations, known as the (scalar)
Hamiltonian constraint and the (3-vector) momentum constraint, constrain the
induced spatial metric $g_{ij}$ and extrinsic curvature $K_{ij}$. The Einstein
constraint equations have been the subject of study for more than fifty years
(cf. [2]). The introduction of the conformal method in the 1970’s as a way of
parameterizing initial data to the Einstein equations led to increased
interest in the development of a complete solution theory for the constraints,
with the theory for constant mean curvature (CMC) spatial slices and closed
manifolds completely developed by 1995. The CMC theory on closed manifolds is
particular satisfying, with nearly all physically interesting cases exhibiting
both existence and uniqueness of solutions.

However, other than the near-CMC result of Isenberg and Moncrief in 1996 [10],
the theory for non-CMC solutions remained completely open until the first far-
from-CMC existence result was establish by Holst et al. [8] in 2008, with
extensions to rough data by Holst et al. in 2009 [9], and to vacuum spacetimes
by Maxwell in 2009 [13]. However, the non-CMC theory remains mostly open, and
what is known is much less satisfying than the CMC case; the new non-CMC
results of Holst et al. and Maxwell are based on new types of topological
fixed point arguments, and while they establish existence, these arguments do
not give uniqueness. Moreover, more recent work of Maxwell on specific
symmetry models show in fact that uniqueness is lost, and also sheds light on
some fundamental problems with the conformal method itself as a
parameterization of initial data on manifolds that are not very close to CMC.

Several decompositions of the equations have been formulated in addition to
the conformal method, although the solution theory for only the conformal
method is well-developed (cf. [3]). In particular, in the extended conformal
thin sandwich (XCTS) decomposition of the constraints, a conformal factor
$\psi$, the lapse $N$, and the shift $\beta_{i}$, are solved for in five
coupled elliptic equations with supplied background data. In parallel with the
mathematical developments in the theory for the conformal method,
computational physicists have uncovered surprising behavior in numerical
solutions to the XCTS formulation. In particular, numerical evidence suggests
the existence of multiple solutions; Pfeiffer and York in [14] numerically
construct two solutions for a specific choice of free data.

In addition, there have been more theoretical results suggesting non-
uniqueness. Under additional assumptions of conformal flatness and time-
symmetry, the Hamiltonian constraint becomes a decoupled equation for the
conformal factor. Baumgarte, Murchadha and Pfeiffer [5] solve this form of the
Hamiltonian constraints using Sobolev functions, to find that an equation
parameter has two admissible values consistent with the constraints. Walsh
[16] performs a Lyapunov-Schmidt analysis (a standard technique in bifurcation
theory) of this form of the Hamiltonian constraint, and shows that at a
critical point with two expected solution branches, the solution branches
should take the form of a quadratic fold. In this article, we examine this
apparent bifurcation phenomena in a methodical way, using modern techniques in
bifurcation theory and in numerical homotopy methods.

Outline of the paper. The remainder of the paper is structured as follows. In
§2, give an overview of the constraint equations in the Einstein system and in
particular the XCTS formulation thereof. In §3, we give a brief introduction
to the mathematical framework for analyzing nonlinear operators in general. We
build on this in §4 where we expound on the theoretical framework underlying
bifurcation analysis. In §5, we develop the main ideas behind the construction
of numerical homotopy, or path-following, methods in the numerical treatment
of bifurcation phenomena, and subsequently in §6 present the methodology of
the numerical techniques we perform for this problem. In §7, we apply the
continuation software package AUTO to the constraint problem, and verify the
presence of the fold with homotopy-based numerical methods. We confirm the
earlier results, as well as provide a framework for a more careful exploration
of the solution theory for various parameterizations of the constraint
equations. Analyzing the Hamiltonian constraint for time-symmetric conformally
flat initial data, as in [16, 5], we demonstrate the existence and location of
a critical point, evidence that the solution branch at the critical point
forms a one-dimensional fold, and the form of the solution as continued past
the critical point.

##  2\. Conformal Thin Sandwich Decomposition

In General Relativity it is common to look at the curvature of spatial
hypersurfaces taken at time-slices of space-time. The Einstein constraint
equations are conditions on the induced spatial metric $g_{ij}$ and the second
fundamental form $K_{ij}$ for being a spatial slice. In addition, there are
six evolution equations that govern how this geometric data evolves in a full
spacetime. In solving the equations, a variety of decompositions have been
proposed. In the XCTS formulation, proposed by York [11], the shift vector
$\beta_{i}$, the lapse $N$ and a conformal factor $\psi$ are solved for given
a set of supplied data that includes
$(\tilde{g}_{ij},\tilde{u}_{ij},K,\partial_{t}K)$, where $\tilde{g}_{ij}$ is
the conformally related induced spatial metric, $\tilde{u}_{ij}$ its time
derivative, and $K$ the trace of the extrinsic curvature. The lapse and shift
are formed from taking a level set of $t$ and looking at the normal to the
hypersurface $n=N^{-1}(\partial_{t}-\beta^{i}\partial_{i})$ where $i$ indexes
over spatial coordinates [4]. The Hamiltonian constraint of the XCTS equations
can be written as

$\tilde{\nabla}^{2}\psi-\frac{1}{8}\tilde{R}\psi-\frac{1}{12}K^{2}\psi^{5}+\frac{1}{8}\psi^{-7}\tilde{A}^{ij}\tilde{A}_{ij}+2\pi\psi^{5}\rho=0$
(2.1)

Here $\tilde{\nabla}$ represents the covariant derivative, $\tilde{R}$ the
trace of the Ricci tensor and $\tilde{A}$ the trace free part of the extrinsic
curvature, all associated with the conformally scaled metric $\tilde{g}_{ij}$.
Following [5, 16] we assume time-symmetry (so $K_{ij}=0$) and conformally flat
initial data. This constraint then reduces to

$\nabla^{2}\psi+2\pi\rho\psi^{5}=0$ (2.2)

Following [5] we let $\rho$ be the constant mass-density of a star. Without
loss of generality, we take $\rho=0$ outside $r=1$ and look for solutions to
(2.2) with boundary conditions

$\displaystyle\frac{\partial\psi}{\partial r}$ $\displaystyle=0,\quad r=0$
(2.3) $\displaystyle\psi$ $\displaystyle=1,\quad r=1$ (2.4)

We can note that the maximum principle does not apply for this equation and
hence uniqueness is not guaranteed. We will see that this equation has two,
one, or no solutions, depending on the value of $\rho$.

##  3\. Nonlinear Operators on Banach Spaces

Let $X$ and $Y$ be Banach spaces, and let $X^{\prime}$ and $Y^{\prime}$ be
their respective dual spaces. Given a (generally nonlinear) map $F\colon X\to
Y$, we are interested in the following general problem:

$\mbox{Find}~{}u\in X~{}\mbox{such~{}that}~{}F(u)=0\in Y.$ (3.1)

We will give a brief overview of techniques for analyzing solutions to (3.1),
and for characterizing their behavior with respect to parameters.

We say that the problem ${F(u)=0}$ is well-posed if there is (a) existence,
(b) uniqueness, and (c) continuous dependence of the solution on the data of
the problem. Recall that if $F$ is both one-to-one (injective) and onto
(surjective), it is called a bijection, in which case the inverse mapping
$F^{-1}$ exists, and we would have both existence and uniqueness of solutions
to the problem ${F(u)=0}$. Recall that $F\colon X\to Y$ is a continuous map
from the normed space $X$ to the normed space $Y$ if
$\lim_{j\to\infty}u_{j}=u$ implies that $\lim_{j\to\infty}F(u_{j})=F(u)$,
where $\\{u_{j}\\}$ is a sequence, $u_{j}\in X$. If both $F$ and $F^{-1}$ are
continuous, then $F$ is called a homeomorphism. If both $F$ and $F^{-1}$ are
differentiable (see the below for the definition of differentiation of
abstract operators in Banach spaces), then $F$ is called a diffeomorphism. If
both $F$ and $F^{-1}$ are $k$-times continuously differentiable, then $F$ is
called a $C^{k}$-diffeomorphism. A linear map between two vector spaces is a
type of homomorphism (structure-preserving map); a linear bijection is called
an isomorphism.

We will need to assemble just a few basic concepts involving general nonlinear
maps in Banach spaces, to provide the mathematical framework for the
discussions in the next section. In particular, the following notion of
differentiation of maps on Banach spaces will be required (see [15, 17] for
more complete discussions).

######  Definition 3.1.

Let $X$ and $Y$ be Banach spaces, let $F\colon X\to Y$, and let $D\subset X$
be an open set. Then the map $F$ is called Fréchet- or F-differentiable at
$u\in D$ if there exists a bounded linear operator $F_{u}(u)\colon X\to Y$
such that:

$\lim_{\|h\|_{X}\to 0}\frac{1}{\|h\|_{X}}\|F(u+h)-F(x)-F_{u}(u)(h)\|_{Y}=0.$

The bounded linear operator $F_{u}(u)$ is called the F-derivative of $F$ at
$u$. The F-derivative of $F$ at $u$ can again be shown to be unique. If the
F-derivative of $F$ exists at all points $u\in D$, then we say that $F$ is
F-differentiable on $D$. If in fact $D=X$, then we simply say that $F$ is
F-differentiable, and the derivative $F_{u}(\cdot)$ defines a map from $X$
into the space of bounded linear maps, $F_{u}\colon X\to\mathcal{L}(X,Y)$. In
this case, we say that $F\in C^{1}(X;Y)$. Many of the properties of the
derivative of smooth functions over domains in $\mathbb{R}^{n}$ carry over to
this abstract setting, including the chain rule: If $X$, $Y$, and $Z$ are
Banach spaces, and if the maps $F\colon X\to Y$ and $G\colon Y\to Z$ are
differentiable, then the derivative of the composition map $H=G\circ F$ also
exists, and takes the form

$H_{u}(u)=(G\circ F)_{u}(u)=G_{F}(F(u))\circ F_{u}(u),$

where $H_{u}\colon X\to\mathcal{L}(X,Z)$, $F_{u}\colon X\to\mathcal{L}(X,Y)$,
and $G_{F}\colon Y\to\mathcal{L}(Y,Z)$. Higher order Fréchet (and Gâteaux)
derivatives can be defined in the obvious way, giving rise to multilinear maps
and giving meaning to the notation $F\in C^{k}(X;Y)$. Note that below we will
often encounter functions of two variables $F(u,\lambda)$, and will be
interested in the derivatives of such functions with respect to each variable;
we will denote these using the consistent notation $F_{u}$ and $F_{\lambda}$.
See [1, 15] for more complete discussions of general maps and differentiation
in Banach spaces.

A fundamental concept concerning linear operators that we will need for the
discussions below is that of a Fredholm operator. Let $X$ and $Y$ be Banach
spaces, and let $A\in\mathcal{L}(X,Y)$, or in other words, $A$ is a bounded
linear operator from $X$ to $Y$. In the case that $X$ and $Y$ have additional
Hilbert space structure, wherein there is an inner-product, the Riesz
Representation Theorem implies that the adjoint operator
$A^{*}\in\mathcal{L}(Y,X)$, defined as the operator for which
$(Ax,y)=(x,A^{*}y)$, exists uniquely. There are four fundamental subspaces of
$X$ and $Y$ associated with $A$ and $A^{*}$, namely:

  1. (i)

$\mathcal{N}(A):$ null space (or kernel) of $A$

  2. (ii)

$\mathcal{R}(A):$ range space of $A$

  3. (iii)

$\mathcal{N}(A^{*}):$ null space (or kernel) of $A^{*}$

  4. (iv)

$\mathcal{R}(A^{*}):$ range space of $A^{*}$

One can consider the dimension (dim) and co-dimension (codim) of each of these
four spaces, where co-dimension is taken to be relative to the larger spaces
that they are subspaces of. The operator $A\in\mathcal{L}(X,Y)$ is called a
Fredholm operator if and only if:

  1. (i)

$\mathrm{dim}(\mathcal{N}(A))<\infty$

  2. (ii)

$\mathrm{dim}(\mathcal{R}(A))<\infty$

The difference of these two dimensions, namely

$\mathrm{ind}(A)=\mathrm{dim}(\mathcal{N}(A))-\mathrm{dim}(\mathcal{R}(A))$
(3.2)

is called the Fredholm index of $A$. A basic result about Fredholm operators
is the following.

######  Theorem 3.2.

If $A\in\mathcal{L}(X,Y)$ is a Fredholm operator, then

  1. (i)

$\mathcal{R}(A)$ is closed.

  2. (ii)

$A^{*}$ is Fredholm with index $\mathrm{ind}(A^{*})=-\mathrm{ind}(A)$.

  3. (iii)

$K$ is a compact operator, then $A+K$ is Fredholm with
$\mathrm{ind}(A+K)=\mathrm{ind}(A)$.

###### Proof.

See [17]. ∎

##  4\. Bifurcation Theory for Nonlinear Operators Equations

Bifurcation Theory studies the branching of solutions as governed by
parameter(s). Throughout this section we will refer to a ”solution curve”
which, for $F(u,\lambda)$ plots points in $(||u||,\lambda)$ space at which
$(u,\lambda)$ solves $F(u,\lambda)=0$, and $||u||$ denotes the norm of $u$.
Specifically, we are interested in the local behavior of the solution curve in
a neighborhood of a known solution $(u_{0},\lambda_{0})$. This is because, in
order to explore the solution space to a certain operator equation
$F(u,\lambda)$, we often solve $F(u,\lambda_{0})$ for $u$, then obtain another
$(u_{1},\lambda_{1})$ from the original solution data we obtain, and similarly
continue to subsequent solutions.

Bifurcation theory has its foundation in the implicit function theorem,

######  Theorem 4.1 (Implicit Function Theorem).

If it holds that, for an operator $F:X\times\mathbb{R}\rightarrow Y$, if
$F(u,\lambda)$ with $F(u_{0},\lambda_{0})=0$, $F$ and $F_{u}$ are continuous
on some region $U\times V$ with $(x_{0},\lambda_{0})\in U\times V$, and
$F_{u}(u_{0},\lambda_{0})$ is nonsingular with a bounded inverse, then there
is a unique branch of solutions $(u(\lambda),\lambda))$ in for $\lambda\in V$,
e.g. $F(u(\lambda),\lambda)=0$. Furthermore $u(\lambda)$ is continuous with
respect to $\lambda$ in $V$

This theorem states that if the operator $F_{u}$ is nonsingular at a certain
point $(x_{0},\lambda_{0})$ there is a unique solution $u$ for each $\lambda$
close to $\lambda_{0}$ on either side of $\lambda_{0}$ and we can plot a one-
dimensional curve in $(||u||,\lambda)$ space through
$(||u_{0}||,\lambda_{0})$. With a singular $F_{u}$, however, such a branch is
not guaranteed, suggesting the possibility of two or more such $u(\lambda)$
branches or no solutions for some $\lambda$ in every neighborhood around
$\lambda_{0}$. The exact form of the branching depends on the dimensions of
$F_{u}(x_{0},\lambda_{0})$, $F_{\lambda}(x_{0},\lambda_{0})$ and whether
$F_{\lambda}(x_{0},\lambda_{0})\in\mathcal{R}(F_{u}(x_{0},\lambda_{0}))$.

In the case of a ”fold”, wherein there is still a one-dimensional path through
$(u_{0},\lambda_{0})$ but the path exists solely on one side of $\lambda_{0}$
for $\lambda$, we have

  1. (i)

$\mathrm{dim}(\mathcal{N}(F_{u}(u_{0},\lambda_{0})))=1$

  2. (ii)

$\mathrm{dim}(\mathcal{N}(F_{u}(u_{0},\lambda_{0})^{*}))=1$, where
$F_{u}(u_{0},\lambda_{0})^{*}$ is the adjoint operator of
$F_{u}(u_{0},\lambda_{0})$

  3. (iii)

$F_{\lambda}(u_{0},\lambda_{0})\notin\mathcal{R}(F_{u}(u_{0},\lambda_{0})$.

It can be shown, under these circumstances, near $(u_{0},\lambda_{0})$ the
continuation of the solution will be of the form

$\displaystyle u(\epsilon)$
$\displaystyle=u_{0}+\epsilon\phi+C_{1}\epsilon^{2}+O(\epsilon^{3})$ (4.1)
$\displaystyle\lambda(\epsilon)$
$\displaystyle=\lambda_{0}+C_{2}\epsilon^{2}+O(\epsilon^{3})$ (4.2)

where $\phi$ is a basis for the null-space of $F_{u}(u_{0},\lambda_{0})$ (see
[6]). Note that there is no linear $\lambda(\epsilon)$ term, so the solution
curve, at first approximation, stays at a constant value of $\lambda$ and
changes in $u$. The specific values of $C_{1}$ and $C_{2}$ are based on the
Hessian of the operator $F$. With a fold the $C_{2}$ will be negative, so the
solution curve shows $\lambda$ increase up to a certain critical point
$\lambda_{c}$ then decreases. With $C_{1}<0$ the actual solution $u$ changes
at first-order along $\phi$, the change then dampens in magnitude. Hence with
$C_{2}\neq 0$, the resulting solution curve appears locally as a sideways
parabola, and hence called a ”simple quadratic fold”. If, on the other hand,
$C_{2}=0$, we have at $(x_{0},\lambda_{0})$ a ”fold of order m” if
$\lambda^{(k)}(\epsilon)=0$ for $k<m$ [12]. In the case of a simple singular
point, we have either

  1. (i)

$\text{dim}(\mathcal{N}(F_{u}(u_{0},\lambda_{0})))=1$, and

  2. (ii)

$F_{\lambda}(u_{0},\lambda_{0})\in\mathcal{R}(F_{u}(u_{0},\lambda_{0}))$,

or we have

  1. (i)

$\text{dim}(\mathcal{N}(F_{u}(u_{0},\lambda_{0})))=2$, and

  2. (ii)

$F_{\lambda}(u_{0},\lambda_{0})\notin\mathcal{R}(F_{u}(u_{0},\lambda_{0}))$.

In this case, we have a situation of branch-switching, in which there are are
two branches of solutions crossing the point $(u_{0},\lambda_{0})$ [12]. With
high-dimensional null-spaces, the situation becomes increasingly complicated,
with multiple branching solutions of various forms. We illustrate the three
representative cases as they would appear in $(||u||,\lambda)$ space in Figure
1.

Figure 1. Common Locally Bifurcating Solution Paths.

We conclude, for completeness, this section with an exposition of the
generalized form of the bifurcation analysis method of Lyapunov-Schmidt, as
used for this problem by Walsh [16]. The exposition follows Zeidler [17]. We
assume that

  1. (i)

$F_{u}(u_{0},\lambda_{0})$ is a Fredholm operator of index $k$

  2. (ii)

$\text{dim}(\mathcal{N}(F_{u}(u_{0},\lambda_{0})))=n$

Then we define projection operators $P:X\rightarrow X$ and $Q:X\rightarrow X$
with $P(X)=\mathcal{N}(F_{u}(u_{0},\lambda_{0}))$ and
$(I-Q)(Y)=\mathcal{R}(F_{u}(u_{0},\lambda_{0}))$. Now the equation
$F(u,\lambda)=0$ is equivalent to the pair

$\displaystyle(I-Q)F(y+z,\lambda)$ $\displaystyle=0$ (4.3) $\displaystyle
QF(y+z,\lambda)$ $\displaystyle=0$ (4.4)

with $y=(I-P)u$ and $z=Pu$. Now the first equation satisfies the assumptions
of the Implicit Function Theorem, and so we can get a unique solution branch
$y(z,\lambda)$, substitute the solution in the second equation to obtain the
branching equation

$QF(y(z,\lambda)+z,\lambda)=0$ (4.5)

then solve for $z(\lambda)$ to get the branch
$u=y(z(\lambda),\lambda)+z(\lambda)$ Note that, in practice, one solves (4.3)
by expanding the operators in the bases of
$\mathcal{N}(F_{u}(u_{0},\lambda_{0}))$ and
$\mathcal{N}(F_{u}(u_{0},\lambda_{0})^{*})$. Hence, to be constructive, one
must already have an estimation of the dimension of the null-space of
$F_{u}(u_{0},\lambda_{0})$. In particular, Walsh [16] performs the analysis
with the starting assumption that
$\text{dim}(\mathcal{N}(F_{u}(u_{0},\lambda_{0})))=1$. So while the technique
is appropriate once this is known, it is not a standalone method of
bifurcation analysis.

##  5\. Numerical Bifurcation Theory

In the procedure of continuation, we seek to find a solution $u$ to a problem
$F(u,\lambda)=0$ as we move along $\lambda$. In the case of a nonsingular
Jacobian of the operator ($F_{u}(u_{0},\lambda_{0})$), it is standard to apply
Newton’s method on a discretization of the solution space. So we perform
continuation by repeatedly iterating $\lambda=\lambda+\Delta\lambda$ then
resolving $F(u,\lambda)=0$ for $u$. However, the procedure is invalid in the
case of a singular Jacobian of the equation operator, necessitating
alternatives for traversing a solution as the parameter varies. Depending on
the form of the bifurcation, various procedures exist. Considering the finite-
dimensional case (e.g. for a discretization of $u$), we have that
$F(u,\lambda)$ maps $\mathbb{R}^{N}\times\mathbb{R}$ to $\mathbb{R}^{N}$. In
this case, the rank of $[F_{u},F_{\lambda}]=N$ if either $F_{u}$ is
nonsingular, which is the case above, or
$F_{\lambda}(u_{0},\lambda_{0})\notin\mathcal{R}(F_{u}(u_{0},\lambda_{0}))$.
In this case $\text{dim}(\mathcal{N}([F_{u},F_{\lambda}]))=1$. Let’s say we
have a solution $F(u_{0},\lambda_{0})=0$. In the latter case, we cannot just
set $\lambda_{1}=\lambda_{0}+\Delta$ (with a constant $\Delta$) and solve for
$u_{1}$, but we can solve for $F(u_{1},\lambda_{1})=0$ together with one more
scalar equation, which we can set to be a constraint on the total magnitude in
the change of $(u,\lambda)$. With pseudo-arclength continuation, at a certain
parameter $\lambda_{0}$ and solution vector $u_{0}$, and a direction vector
$(\dot{u}_{0},\dot{\lambda}_{0})$ of the solution branch determined thus far,
you run Newton’s method on the two equations $F(u_{1},\lambda_{1})=0$ and
$(u_{1}-u_{0})^{*}\dot{u}_{0}+(\lambda_{1}-\lambda_{0})\dot{\lambda}-\Delta
s=0$, where $\Delta s$ is a constant ”arclength” term. It can be shown that
the Newton’s method Jacobian matrix is nonsingular if the point is either one
at which $F_{u}$ is nonsingular or a fold [6].

If $\text{dim}(\mathcal{N}(F_{u}))>1$ or
$F_{\lambda}(u_{0},\lambda_{0})\in\mathcal{R}(F_{u}(u_{0},\lambda_{0}))$ then
the Jacobian of Newton’s method in the pseudo-arclength continuation equations
is singular as well, and other procedures must be used to continue the
solution. Methods of continuation usually involve constructive techniques,
wherein the nullspace vectors for $F_{u}$ and the solution
$F_{u}\phi_{r}=F_{\lambda}$ are explicitly found and coefficients constructed
(for a list of relevant algorithms, see [12]). With even higher-dimensional
null-spaces, this framework is generalized.

##  6\. Setup for Hamiltonian Constraint Bifurcation

We reprint the equations (2.2)–(2.3) here

$\displaystyle\nabla^{2}\psi+2\pi\rho\psi^{5}$ $\displaystyle=0$ (6.1)
$\displaystyle\frac{\partial\psi}{\partial r}$ $\displaystyle=0,\quad r=0$
(6.2) $\displaystyle\psi$ $\displaystyle=1,\quad r=1$ (6.3)

and perform a reduction for bifurcation analysis following [6].

The goal is to transform this equation into a form wherein its linearization
becomes an eigenvalue problem. This can be done by parameterizing an extra
boundary condition. The equation can be rewritten in an equivalent form as

$\displaystyle\nabla^{2}\psi+2\pi\rho\psi^{5}$ $\displaystyle=0$ (6.4)
$\displaystyle\psi(0)$ $\displaystyle=p$ (6.5)
$\displaystyle\frac{\partial\psi}{\partial r}(0)$ $\displaystyle=0$ (6.6)

Now we seek to solve $F(p,\rho)\equiv\psi(1,p,\rho)-1=0$. Writing
$\psi_{p}(t,p,\rho)=\frac{d\psi}{dp}(t,p,\rho)$,
$F_{p}(p,\rho)=\psi_{p}(1,p,\rho)$ and similarly
$F_{\rho}(p,\rho)=\psi_{\rho}(1,p,\rho)$ Now $\psi_{p}$ satisfies

$\displaystyle\nabla^{2}\psi_{p}+10\pi\rho\psi^{4}\psi_{p}$ $\displaystyle=0$
(6.7) $\displaystyle\psi_{p}(0)$ $\displaystyle=1$ (6.8)
$\displaystyle\frac{\partial\psi_{p}}{\partial r}(0)$ $\displaystyle=0$ (6.9)

Which is an eigenvalue problem with a distinct solution. Define
$a(p,\lambda)=\psi_{p}(1,p,\rho)=F_{p}(p,\rho)$. Similarly $\psi_{\rho}$
satisfies

$\displaystyle\nabla^{2}\psi_{\rho}+2\pi\psi^{5}+10\pi\rho\psi^{4}\psi_{\rho}$
$\displaystyle=0$ (6.10) $\displaystyle\psi_{\rho}(0)$ $\displaystyle=0$
(6.11) $\displaystyle\frac{\partial\psi_{\rho}}{\partial r}(0)$
$\displaystyle=0$ (6.12)

and likewise define $b(p,\rho)=\psi_{\rho}(1,p,\rho)=F_{\rho}(p,\rho)$. Now we
have $F_{(p,\rho)}=(a,b)$, So $N(F_{(p,\rho)}))$ can be $\begin{pmatrix}1\\\
0\end{pmatrix}$, $\begin{pmatrix}0\\\ 1\end{pmatrix}$, both, or neither. Let
us discuss each of these cases separately. If the null-space is empty, then we
have a nonsingular Jacobian of the operator, so continuation can proceed with
Newton’s method as standard. If the null-space is $\begin{pmatrix}0\\\
1\end{pmatrix}$, so $b(p,\rho)=0$, then this is a situation where the same
solution exists for a differential increase in $\rho$, which is also an
uninteresting case. If the null-space is $\begin{pmatrix}1\\\ 0\end{pmatrix}$,
so $a(p,\rho)=0$ and $b(p,\rho)\neq 0$, we now have a case where $N(F_{p})=1$
but $F_{\rho}\notin F_{p}$, so we have a fold. Pseudo-arclength continuation
must be performed, and the form of the continuation gleamed from the
coefficients. Finally, in the case of a full two-dimensional null-space, we
have reached another solution branch. Here, there are two solution branches
intersecting, and so we have a choice between three directions in the
continuation of the solution. We now seek to perform this analysis numerically
to locate any critical points for (6.1) and see one of these scenarios occurs.

##  7\. Numerical Results

Recall that a continuation procedure attempts to solve $F(u,\lambda)$ for
varying $\lambda$ by a series of successive iterations along discrete steps of
$\lambda$. The procedure begins at some $\lambda_{0}$, finds the solution
$F(u_{0},\lambda_{0})$, finds the nullspace information of
$F_{u}(u_{0},\lambda_{0})$ and $F_{\lambda}(u_{0},\lambda_{0})$, then using
this information performs a suitable continuation step to find the next
solution $F(u_{1},\lambda_{1})$. We used the continuation software AUTO to
trace the solutions to (2.2) with boundary conditions (2.3). AUTO ([7])
performs numerical continuation on ODEs (notice that due to symmetry the
differential equation becomes an ODE with just variable $r$). AUTO calculates
the dimension of the null-space of the Jacobian of the differential operator
and performs, as required, Newton’s method, pseudo-arclength continuation or
explicit calculation of the null-space directions if it identifies higher-
order bifurcations, as discussed in (5). It uses the bordering algorithm for
the case of pseudo-arclength continuation ([6]) and otherwise more intricate
linear algebraic algorithms for higher-dimensional bifurcations.

Figure 2. Solution curve for (6.1).

Continuation reveals a quadratic fold at a value of $\rho$ around
$\rho_{c}\approx 0.35$. For $\rho<\rho_{c}$ there are two solutions to (2.2),
at $\rho=\rho_{c}$, there is exactly one solution, and at $\rho>\rho_{c}$
there are no solutions. AUTO finds that at $\rho=\rho_{c}$ the nullspace of
the discrete Jacobian differential operator has a dimension of one. In
addition, it finds that the 2nd derivative information indicates that the
continuation has a quadratic form. At all other points along the solution
curve, the Jacobian is nonsingular. Now we let the exponent vary. Writing the
equation as $\nabla^{2}\psi+2\pi\rho\psi^{a}=0$, we investigate the
continuation for different values of $a$. Knowing that the Laplacian operator
on its own has an invertible Jacobian, we expect a fold to appear for some
value of $a$. We find that indeed, with $a>1$ a fold appears, the curvature of
the solution continuation becoming sharper with increasing $a$. In Figure 3 we
see the bifurcation diagram for $a=1,1.25,5,10$.

Figure 3. Bifurcation diagram for $a=1,1.25,5,10$.

We’d like to note that, in contrast to the previous numerical evidence for
non-uniqueness [14], rather than looking at the form of the solution curve and
making an educated guess at the presence of a second solution curve, we
confirm that there is one and only one additional solution curve which
connects to the primary solution set in a quadratic curve and there are no
additional branches or any solutions past $\rho_{c}$.

Now we look at two representative solutions from the two branches. Taking the
two solutions at $\rho=0.2$ for the original problem $a=5$, we see

Figure 4. Solutions for $a=5$, $\rho=0.2$.

We note that this confirms the results of [5], as the solutions have the form
of a Sobolev function and furthermore the conformal factor is considerably
larger, suggesting a higher ADM energy, for the upper branch solutions.

##  8\. Discussion

The Conformal Thin Sandwich approach has been a promising formulation of the
Einstein constraints. The XCTS method appears to have the benefit of more
physically intuitive and less computationally onerous specification of free
data In contrast to its namesake predecessors, the conformal method and thin
sandwich approach. In particular, you save the task of having to specify a
divergence-free part of a symmetric tracefree tensor in the conformal method.
As such it has been used extensively in the construction of the geometry of
binary neutron stars and black holes (see the bibliography 5-12 in [14]). As
such this paper provides a point of caution for numerical relativists.

As Baumgarte et al. [5] noted, the upper branch corresponds to a higher level
of ADM energy than the lower branch in this model of a constant density black
hole. As such, in most applications with this set of asymptotically flat,
spherically symmetric free data, the lower branch should be chosen as the
correct, physically representative solution, but numerical relativists should
be aware that a solver could reach either one and be able to identify the need
to switch to the other branch. In addition, as $\rho\rightarrow\rho_{c}$
standard numerical procedures will become increasingly ill-conditioned.
Pinning the solution to the expected branch is recommended as a solution
strategy.

Finally, it should be noted that since this bifurcation analysis is complete,
rather than simply constructive, and so demonstrates conclusively that 1)
there are exactly two solutions for $\rho<\rho_{c}$ and 2) there are no
solutions for $\rho>\rho_{c}$ for this choice of free data. The fact that the
same choice of free data is consistent with two qualitatively disparate
geometries suggests future investigation in the relation between the structure
in the free and dependent set of data. In addition, it would be an interesting
investigation as to why a higher density is inconsistent with time-symmetry,
conformal flatness and spherical symmetry.

##  9\. Conclusion

In this article, we have examined the apparent bifurcation phenomena in the
XTCS formulation of the Einstein constraints in a methodical way, using modern
techniques in bifurcation theory and in numerical homotopy methods. We first
gave an overview of the Einstein constraints in §2, and followed this in §3
with a brief introduction to the mathematical foundations for, and in §4 the
framework for analyzing bifurcation phenomena in nonlinear operators
equations. In §5, we developed the main ideas behind the construction of
numerical homotopy, or path-following, methods in the numerical treatment of
bifurcation phenomena, and in §6 we presented the set up of and in §7 we
applied the continuation software package AUTO to the constraint problem. We
verified the presence of the fold with homotopy-based numerical methods,
confirming the earlier results. Analyzing the Hamiltonian constraint for time-
symmetric conformally flat initial data, as in [16, 5], we demonstrated the
existence and location of a critical point, evidence that the solution branch
at the critical point is one-dimensional, and the form of the solution as
continued past the critical point is a simple quadratic fold. We confirm
Walsh’s [16] constructive Lyapunov-Schmidt analysis by showing numerically
that there is indeed a solution point at which there is a one-dimensional
kernel for the linearization of the differential equation operator, justifying
the assumption made in his analysis, as well as confirming numerically that
indeed the form of the solution continuation expansion coefficients are such
that this equation exhibits a quadratic fold rather than branching or higher-
order folds.

The techniques presented here can be viewed as providing a framework for a
more careful exploration of the solution theory for various parameterizations
of the constraint equations, as well as the geometric relationship between the
free data in the model problem investigated.

##  10\. Acknowledgments

MH would like to express his appreciation to Herb Keller for introducing him
to the continuation methodology and techniques used in the paper, while MH was
a postdoc with Professor Keller at Caltech. Professor Keller was very
enthusiastic about eventually doing a careful numerical bifurcation analysis
of the Einstein constraint equations. MH would also like to thank Kip Thorne
for the many conversations about mathematical and numerical general
relativity.

MH was supported in part by the NSF through Awards 0715146 and 0915220, and by
DOD/DTRA through Award HDTRA-09-1-0036. VK was supported in part by NSF Award
0715146 and 0915220.

## References

  * [1] R. Abraham, J. E. Marsden, and T. Ratiu.  Manifolds, Tensor Analysis, and Applications.  Springer-Verlag, New York, NY, 1988.
  * [2] R. Arnowitt, S. Deser, and C. Misner.  The dynamics of general relativity.  In L. Witten, editor, Gravitation, pages 227–265. Wiley, New York, NY, 1962.
  * [3] R. Bartnik and J. Isenberg.  The constraint equations.  In P. Chruściel and H. Friedrich, editors, 50 Years of the Cauchy Problem, in honour of Y. Choquet-Bruhat, 2002 Cargese Meeting, 2003\.  to appear.
  * [4] R. Bartnik and J. Isenberg.  The Einstein equations and the large scale behavior of gravitational fields, chapter The Einstein Constraints.  Birkhauser Verlag, 2004.
  * [5] T. W. Baumgarte, N. O. Murchadha, and H. P. Pfeiffer.  Einstein constraints: Uniqueness and nonuniqueness in the conformal thin sandwich approach.  Phys. Rev. D, 75(044009), 2007.
  * [6] E. J. Doedel.  Numerical analysis of bifurcation problems.  Spring School on Numerical Software, 1997.
  * [7] T. F. F. Y. A. K. B. S. X.-J. W. E. J. Doedel, A. R. Champneys.  Auto97: Continuation and bifurcation software for ordinary differential equations (with homcont), 1997.  Description available at: http://cmvl.cs.concordia.ca/publications/auto97.ps.gz.
  * [8] M. Holst, G. Nagy, and G. Tsogtgerel.  Far-from-constant mean curvature solutions of Einstein’s constraint equations with positive Yamabe metrics.  Phys. Rev. Lett., 100(16):161101.1–161101.4, 2008.  Available as arXiv:0802.1031 [gr-qc].
  * [9] M. Holst, G. Nagy, and G. Tsogtgerel.  Rough solutions of the Einstein constraints on closed manifolds without near-CMC conditions.  Comm. Math. Phys., 288(2):547–613, 2009.  Available as arXiv:0712.0798 [gr-qc].
  * [10] J. Isenberg and V. Moncrief.  A set of nonconstant mean curvature solutions of the Einstein constraint equations on closed manifolds.  Class. Quantum Grav., 13:1819–1847, 1996.
  * [11] J. James W. York.  Conformal ’thin-sandwich’ data for the initial-value problem of general relativity.  Phys. Rev. Lett., 82(7), 1999.
  * [12] H. B. Keller.  Numerical Methods in Bifurcation Problems.  Tata Institute of Fundamental Research, Bombay, India, 1987.
  * [13] D. Maxwell.  A class of solutions of the vacuum Einstein constraint equations with freely specified mean curvature.  Math. Res. Lett., 16(4):627–645, 2009.
  * [14] H. P. Pfeiffer and J. James W. York.  Uniqueness and nonuniqueness in the einstein constraints.  Phys. Rev. Lett., 95(091101), 2005.
  * [15] I. Stakgold and M. Holst.  Green’s Functions and Boundary Value Problems.  John Wiley & Sons, Inc., New York, NY, third edition, 888 pages, February 2011.  The preface and table of contents of the book are available at: http://ccom.ucsd.edu/~mholst/pubs/dist/StHo2011a-preview.pdf.
  * [16] D. M. Walsh.  Non-uniqueness in conformal formulations of the einstein constraints.  Class. Quantum Grav., 24(8), 2007.
  * [17] E. Zeidler.  Nonlinear Functional Analysis and its Applications, volume I: Fixed Point Theorems.  Springer-Verlag, New York, NY, 1991.