# Adaptation of Biomedical and Clinical Pretrained Models to French Long
Documents: A Comparative Study

Adrien Bazoge1,2 Emmanuel Morin2
Béatrice Daille2 Pierre-Antoine Gourraud1
1Nantes Université, CHU Nantes, Clinique des données, INSERM, CIC 1413,
F-44000 Nantes, France
2Nantes Université, École Centrale Nantes, CNRS, LS2N, UMR 6004, F-44000
Nantes, France
<EMAIL_ADDRESS>
###### Abstract

Recently, pretrained language models based on BERT have been introduced for
the French biomedical domain. Although these models have achieved state-of-
the-art results on biomedical and clinical NLP tasks, they are constrained by
a limited input sequence length of 512 tokens, which poses challenges when
applied to clinical notes. In this paper, we present a comparative study of
three adaptation strategies for long-sequence models, leveraging the
Longformer architecture. We conducted evaluations of these models on 16
downstream tasks spanning both biomedical and clinical domains. Our findings
reveal that further pre-training an English clinical model with French
biomedical texts can outperform both converting a French biomedical BERT to
the Longformer architecture and pre-training a French biomedical Longformer
from scratch. The results underscore that long-sequence French biomedical
models improve performance across most downstream tasks regardless of sequence
length, but BERT based models remain the most efficient for named entity
recognition tasks.

##  1 Introduction

Recent pretrained language models based on the Transformer architecture
Vaswani et al. (2017) have succeeded in improving the field of natural
language processing, enabling state-of-the-art performance across a broad
range of tasks and forming the backbone of contemporary deep learning
architectures. Transformer-based models have been adapted to various domains,
including biomedical and clinical. Initially, these adaptations used
pretrained language models (PLMs) based on BERT Devlin et al. (2019) like
BioBERT Lee et al. (2019), ClinicalBERT Huang et al. (2019), and PubMedBERT Gu
et al. (2021). Recently, there has been a notable shift towards the use of
Large Language Models (LLMs) with models like Med-PaLM 2 Singhal et al.
(2023), PMC-LLaMA Wu et al. (2023), and MedAlpaca Han et al. (2023). Although
LLMs have drawn the attention in recent years, several studies suggest that
PLMs remain more suitable than LLMs for traditional NLP tasks, such as named
entity recognition and text classification He et al. (2023); Ramachandran et
al. (2023); Labrak et al. (2023c); Zhou et al. (2023); Yuan et al. (2023).

Due to the full self-attention mechanism, BERT-based models suffers from the
limited input sequence length of 512 tokens when applied to clinical notes Gao
et al. (2021). Indeed, clinical documents are generally large documents that
can contain up to several thousand words. To address the constraints of
modeling long texts with PLMs, the input sequence can be truncated to the
initial 512 tokens or processed at the document-level using a sliding window
of 512 tokens, with optional overlap. However, such methods fail to consider
long-range dependencies beyond 512 tokens, as they lack a self-attention
mechanism that operates across segments processed in parallel. Recently,
various works have adapted the transformers’ attention mechanism for long
sequences by reducing its complexity Tay et al. (2021). Some of these models
were then adapted to the clinical domain for English Li et al. (2023).

In this paper, we explore the adaptation of long-sequences pretrained models
for French biomedical and clinical domains. The contributions of this article
are the following:

  * •

The introduction of French pretrained language models based on the Longformer
architecture for processing long texts in the biomedical and clinical domains.
All models are freely available under the open-source Apache 2.0
license111https://huggingface.co/abazoge/DrLongformer.

  * •

A comparative study to analyze different pre-training strategies for the
Longformer architecture. Our results demonstrate that further pre-training an
English clinical model with French biomedical texts can yield better
performance than converting a BERT model to the Longformer architecture or
pre-training a French biomedical Longformer from scratch.

  * •

We demonstrate that our long-sequence French biomedical models improve
performance on a majority of biomedical and clinical tasks, regardless of
sequence length, but BERT models remain the most efficient for NER tasks.

##  2 Related works

###  2.1 Attention mechanism for long-sequence transformers

The self-attention mechanism in Transformers is a key component of recent
pretrained language models. It allows each token in an input sequence to
interact independently and in parallel with all other tokens. However, the
computational and memory demands of the Transformer’s self-attention grow
quadratically with the sequence length. This makes processing long sequences
computationally expensive. Consequently, models like BERT and its variants are
limited to input sequence lengths of 512 tokens, which can be limiting when
working with long documents for specific tasks.

Various adaptation of the Transformers’ self-attention mechanism have been
proposed to reduce this complexity Tay et al. (2021, 2022). One of the
possible approaches is sparse attention and involves using non-full attention
patterns to reduce the scope of attention. The sparse attention mechanisms
implemented in various pretrained models often comprise several atomic
patterns, including global attention, band attention (such as sliding window
or local attention), dilated attention, random attention, and block local
attention.

The Sinkhorn attention Tay et al. (2020) combines a content-based block sparse
attention along with block local attention. Longformer Beltagy et al. (2020)
rely on a sliding window and a global attention. In addition to sliding window
and a global attention, BigBird Zaheer et al. (2020) use an additional random
attention to approximate full attention. More recently, the LSG (Local Sparse
Global) attention was introduced Condevaux and Harispe (2023). This attention
relies on three components: block local attention, sparse attention to capture
extended context and global attention. All these attention mechanisms allow
reducing the attention complexity from $O(n^{2})$ to $O(nlogn)$ or $O(n)$.
Many other methods can improve the efficiency by replacing the self-attention
matrix by a low-rank approximation, such as Linformer Wang et al. (2020),
Performer Choromanski et al. (2020a, b), Random Feature Attention Peng et al.
(2021), and LUNA Ma et al. (2021).

###  2.2 Biomedical and clinical pretrained models

BERT-based models Devlin et al. (2019) have been extensively adapted for the
biomedical and clinical domains. In English, various pretrained models have
been introduced for the biomedical domain, such as BioBERT Lee et al. (2019),
PubMedBERT Gu et al. (2021), and SciBERT Beltagy et al. (2019). These models
are primarily trained on scientific articles from sources like PubMed or
Semantic Scholar. In the clinical domain, models like BlueBERT Peng et al.
(2019) and ClinicalBERT Huang et al. (2019) stand out, both of which are
trained on clinical narratives from the MIMIC database Johnson et al. (2016).

For languages other than English, the adaptation has been less extensive but
nonetheless noteworthy. Models have been proposed for German Bressem et al.
(2023), Portuguese Schneider et al. (2020), Spanish Carrino et al. (2021), and
Turkish Turkmen et al. (2022). For French, multiple models have been
introduced recently: AliBERT Berhe et al. (2023), CamemBERT-Bio Touchent et
al. (2023), and DrBERT Labrak et al. (2023a). They leverage biomedical data
from various sources to pre-train their models, such as scientific articles,
drug leaflets, and clinical cases.

For long documents, only the Longformer and BigBird models have been adapted
with continual pre-training using the MIMIC database Johnson et al. (2016),
resulting in the Clinical-Longformer Li et al. (2023) and Clinical-BigBird Li
et al. (2023) models, respectively. Beside English, no model has been adapted
for other languages.

##  3 Models pre-training

In this section, we describe the pre-training modalities of our models. We
first introduce the pre-training strategies for the models’ adaptation in
Section 3.1. In Section 3.2, we describe the dataset used for the models’ pre-
training. Finally, in Section 3.3, we provide the technical details of the
models’ pre-training.

###  3.1 Pre-training strategies

Prior studies on pre-training French biomedical language models have drawn
different conclusions about the impact of pre-training strategies on
downstream task performance. Labrak et al. (2023a) showed with DrBERT that
pre-training a BERT model from scratch is more effective than domain
adaptation through continual pre-training from CamemBERT’s weights, a French
general model. On the other hand, Touchent et al. (2023) demonstrate on NER
tasks that CamemBERT-bio, a biomedical model also pretrained from the weights
of CamemBERT, outperform DrBERT when evaluated with metrics other than macro
F1. These different conclusions suggest a reassessment of the findings.

In this study, we evaluate various pre-training strategies to adapt the
Longformer architecture to the French biomedical domain and introduce three
models:

  * •

DrBERT-4096: Converting DrBERT into the Longformer architecture by replacing
the weights of Longformer with those of DrBERT.

  * •

DrLongformer-FS: Pre-training a model from scratch, including the subword
tokenizer built from the pre-training dataset.

  * •

DrLongformer-CP: Continuing the pre-training of the clinical English model
Clinical-Longformer on French biomedical data, while keeping the initial
tokenizer.

Only English models can be used as a starting point for the continual pre-
training strategy since no Longformer model is available for French. All pre-
training scripts for each strategy are available to reproduce the
experiments222https://github.com/abazoge/DrLongformer.

###  3.2 Datasets

For the pre-training of the models, we used NACHOS Labrak et al. (2023a), a
corpus containing a wide range of biomedical data crawled from the web. In
total, the pre-training dataset consists of approximately 1.276 billion words,
corresponding to 2,367,419 documents. Each document was segmented into subword
units using the tokenizer specific to each model, and then split into
sequences of 4,096 subword units. Sequences at the end of documents that were
shorter than 512 tokens were not retained. Those longer than 512 tokens were
padded up to 4,096 tokens.

For the model pretrained from scratch, named DrLongformer-FS, we use a
WordPiece Wu et al. (2016) tokenizer built from the pre-training dataset, with
a vocabulary size of 50,265 tokens. For the DrBERT-4096 and DrLongformer-CP
models, the tokenizer is the same as DrBERT and Clinical-Longformer models
respectively.

###  3.3 Optimization & Pre-training

All pretrained models are based on the Longformer architecture (12 layers, 768
hidden dimensions, 12 attention heads, 149M parameters). The models are
trained on the Masked Language Modeling (MLM) task using HuggingFace library
Wolf et al. (2019) with a masking probability of 15%.

For the DrLongformer-FS and DrLongformer-CP models, we performed around 50k
steps with a batch size of 256, each sequence being filled with 4,096 tokens,
making it possible to process around one million tokens per step. The learning
rate is gradually increased linearly over 10k steps, from zero to the initial
learning rate of 5$\times$10-5. Models are trained on 128 Nvidia V100 32 GB
GPUs for 20 hours. We also use mixed-precision training (FP16) Micikevicius et
al. (2017) to reduce the memory footprint.

After adapting the DrBERT model to the Longformer architecture, resulting in
the new model named DrBERT-4096, we pretrained it for an additional epoch on
NACHOS, since DrBERT had already undergone pretraining on NACHOS. We performed
13,300 steps over 22 hours using a single Tesla A100 GPU with a batch size of
8 and gradient accumulation over 4 steps, resulting in an effective batch size
of 32.

Dataset | Task | Metric | Train | Validation | Test |  | Avg. seq.
---
length
| Max seq.
---
length
Public datasets |  |  |  |  |  |
ESSAI | POS Tagging | seqeval F1 | 5,072 | 725 | 1,450 | 22.55 | 135
CAS Corpus | POS Tagging | seqeval F1 | 2,653 | 379 | 758 | 23.07 | 136
CLISTER | Semantic textual similarity | EDRM/Spearman | 499 | 101 | 400 | 25.21 | 103
DEFT-2020 | Semantic textual similarity | EDRM/Spearman | 498 | 102 | 410 | 45.14 | 118
DEFT-2020 | Multi-class Classification | Weighted F1 | 460 | 112 | 530 | 90.19 | 254
DEFT-2021 | Multi-label Classification | Weighted F1 | 118 | 49 | 108 | 404.78 | 1,710
DEFT-2021 | NER | seqeval F1 | 121 | 46 | 108 | 404.78 | 1,710
QUAERO - EMEA | NER | seqeval F1 | 13 | 12 | 15 | 1,006.42 | 1,549
E3C - Clinical | NER | seqeval F1 | 146 | 22 | 81 | 356.39 | 714
E3C - Temporal | NER | seqeval F1 | 56 | 8 | 17 | 354.86 | 685
FrenchMedMCQA | MCQA | EMR/Hamming Score | 2,171 | 312 | 622 | 44.80 | 174
FrenchMedMCQA | Multi-class Classification | Weighted F1 | 2,171 | 312 | 622 | 44.80 | 174
MorFITT | Multi-label Classification | Weighted F1 | 1,514 | 1,022 | 1,088 | 226.33 | 1 425
DiaMed | Multi-class Classification | Weighted F1 | 509 | 76 | 154 | 329.92 | 1,379
Private datasets |  |  |  |  |  |
aHF classification | Binary Classification | Weighted F1 | 1,179 | 132 | 328 | 782.77 | 4,142
aHF NER | NER | seqeval F1 | 358 | 51 | 103 | 929.78 | 1,992

Table 1: Description, metrics and statistics of downstream biomedical and
clinical NLP tasks. The average and max sequence length are words length.

##  4 Downstream tasks evaluation

We evaluate the long-sequence models on 16 downstream tasks. First, we
describe the datasets and their associated tasks in Section 4.1. The
performances of our models are compared to state-of-the-art French short-
sequence biomedical models and English long-sequence models, which are
presented in Section 4.2. Finally, in Section 4.3, we detail the fine-tuning
procedure of the models on the downstream tasks along with the evaluation
metrics. All fine-tuning scripts are available to reproduce the
experiments333https://github.com/DrBenchmark/DrBenchmark.

###  4.1 Evaluation tasks

The evaluation primarily focuses on datasets with long sequences, i.e.,
sequences longer than 250 words. The sequence length in tokens depends on the
tokenizer used. With an average tokenization of a word into two tokens, a
sequence longer than 250 words would exceed the capacity of a short-sequence
model like BERT. Therefore, long-sequence models are necessary to process such
sequences with long-term dependencies. Based on this selection criteria, the
following biomedical datasets from DrBenchmark Labrak et al. (2024) were
included in the evaluation: DEFT-2021, DiaMed, E3C, QUAERO-EMEA, MorFITT and
the classification task from the DEFT-2020 dataset. It is important to note
that DrBenchmark’s collection does not cover datasets derived directly from
clinical practice. To address this gap and enhance the diversity of our data
sources, we supplemented our evaluation with two tasks from an in-house
clinical dataset focused on acute heart failure. This initial set of tasks
primarily consisted of document classification and NER. To add more diversity
in the evaluation tasks, we add POS tagging (CAS corpus and ESSAI), semantic
textual similarity (CLISTER and DEFT-2020) and multiple-choice question
answering (FrenchMedMCQA) tasks.

The descriptions and statistics of all datasets are presented in Table 1, and
each dataset is further described below.

#### ESSAI

Dalloux et al. (2021) is a corpus of 7,247 sentences taken from French
clinical trials and clinical cases, annotated in 41 POS tags. The dataset was
randomly split into 3 subsets of 70%, 10% and 20% of the total data for
training, validation and test respectively.

#### CAS corpus

Grabar et al. (2018) contains sentences extracted from clinical cases that
have been annotated for POS tagging with 31 classes. The dataset was randomly
split into 3 subsets of 70%, 10% and 20% of the total data for training,
validation and test respectively.

#### CLISTER

Hiebel et al. (2022) is a French clinical semantic textual similarity (STS)
dataset of 1,000 sentence pairs manually annotated by several annotators, who
assigned similarity scores ranging from 0 to 5 to each pair. The scores were
then averaged together to obtain a floating-point number representing the
overall similarity.

#### DEFT-2020

Cardon et al. (2020) contains clinical cases introduced in the 2020 edition of
French Text Mining Challenge, called DEFT. The dataset is annotated for two
tasks: (i) semantic textual similarity and (ii) multi-class classification.
The first task aims at identifying the degree of similarity within pairs of
sentences, from 0 (the less similar) to 5 (the most similar). The second task
consists in identifying, for a given sentence, the most similar sentence among
three sentences provided.

#### DEFT-2021

Grouin et al. (2021) is a subset of 275 clinical cases taken from the 2019
edition of DEFT. This dataset is manually annotated in two tasks: (i) multi-
label classification and (ii) NER. The multi-label classification task is
annotated with 23 axes derived from Chapter C of the Medical Subject Headings
(MeSH). The second task focuses on fine-grained information extraction for 13
entities, such as diseases, signs, or symptoms mentioned in the clinical
cases.

#### QUAERO corpus

Névéol et al. (2014) contains annotated entities and concepts for NER tasks.
The dataset covers two text genres: EMEA, containing drug leaflets, and
MEDLINE, containing scientific article titles, but only the EMEA subset was
used in this study. The 10 annotated entities correspond to the UMLS Semantic
Groups Lindberg et al. (1993). Due to the presence of nested entities, we
simplified the evaluation process by retaining only annotations at the higher
granularity level from the BigBio Fries et al. (2022) implementation,
following the approach described in Touchent et al. (2023). Owing to this
choice translates into an average loss of 6.06% of the annotations on the EMEA
subset.

#### E3C

Magnini et al. (2020) is a multilingual dataset of clinical narratives
annotated for the NER task. It consists of two types of annotations: (i)
clinical entities (e.g., pathologies), (ii) temporal information and
factuality (e.g., events). While the dataset covers 5 languages, only the
French portion is used in this study. The dataset consists of two layers:
layer 1 is manually annotated while layer 2 is semi-automatically annotated.
We randomly split the dataset with 70% for training, 10% for validation, and
20% for testing, as explained in Table 2.

Subset | Train | Validation | Test
---|---|---|---
Clinical | 87.38 % of layer 2 | 12.62 % of layer 2 | 100 % of layer 1
Temporal | 70 % of layer 1 | 10 % of layer 1 | 20 % of layer 1
Table 2: Description of the sources for the E3C subsets.

#### FrenchMedMCQA

Labrak et al. (2022) is a Multiple-Choice Question Answering (MCQA) dataset
for biomedical domain. It contains 3,105 questions coming from real exams of
the French medical specialization diploma in pharmacy, integrating single and
multiple answers. The first task consists of automatically identifying the set
of correct answers among the 5 proposed for a given question. The second task
consists of identifying the number of answers (between 1 and 5) supposedly
correct for a given question.

#### MorFITT

Labrak et al. (2023b) is a multi-label dataset that has been annotated with
specialties from the medical field. The dataset contains 3,624 abstracts from
PMC Open Access. It has been annotated across 12 medical specialties, for a
total of 5,116 annotations.

#### DiaMed

is a dataset of 739 clinical cases that have been manually annotated by
several annotators, one of which is a medical expert, with 22 chapters of the
International Classification of Diseases, 10th Revision (ICD-10). These
chapters provide a general description of the type of injury or disease.

#### Acute heart failure (aHF) classification

This task consists of the classification of hospital stays reports according
to the presence or absence of a diagnostic of acute heart failure. This corpus
consists of 1,639 hospital stays reports from Nantes university hospital,
which are labeled as positive or negative to acute heart failure.

#### Acute heart failure (NER)

This corpus contains 350 hospital stay reports from Nantes University
Hospital. The reports are annotated with 46 entity types related to the
following clinical information: cause of chronic heart failure, triggering
factor for acute heart failure, diabetes, smoking status, heart rate, blood
pressure, weight, height, medical treatment, hypertension and left ventricular
ejection fraction. Overall, the corpus contains 6,116 clinical entities.

###  4.2 Baselines models

We describe the short and long-sequence pretrained models used as baselines in
this study.

#### DrBERT

Labrak et al. (2023a) is a French biomedical language model built using a
from-scratch pre-training of the RoBERTa architecture on NACHOS, a French
public biomedical corpus integrating 1.08 billion words (7.4 GB).

#### CamemBERT-bio

Touchent et al. (2023) is a French biomedical language model pretrained from
the weights of CamemBERT Martin et al. (2020) on the French public corpus
biomed-fr made of 413 million words (2.7 GB) and containing a wide range of
data taken from the web.

#### Clinical-Longformer

Li et al. (2023) is a clinical language model pretrained from the weight of
the Longformer model on approximately 2 million clinical notes extracted from
the MIMIC-III database Johnson et al. (2016). This model support input
sequences up to 4,096 tokens long, which correspond to 8 times the input
sequence limit of BERT models.

###  4.3 Fine-tuning and evaluation metrics

All the models are fine-tuned regarding a strict protocol using the same
hyperparameters for each downstream task. On every task, the batch size is 2
for Longformer based models, and 16 or 32 for BERT based models. To ensure
robustness and reliability in the evaluation, we experiment multiple learning
rate: {1$\times$10-4, 1$\times$10-5, 2$\times$10-5, 5$\times$10-5}. For each
task, four runs are conducted for each learning rate, and we retain the
results of the learning rate that achieves the best average over the four
runs. Finally, depending on the tasks and the models’ architecture, the fine-
tuning protocol and the evaluation metrics can vary.

#### Named entity recognition and POS tagging

For token classification tasks such as POS tagging and NER, a token
classification head (a linear layer) is added on top of the hidden-states
output. BERT based models are fine-tuned and evaluated at sentence-level,
whereas Longformer models are fine-tuned at document-level (or half document
when it exceeds 4,096 tokens). The models’ performance is evaluated with the
seqeval Nakayama (2018) metric in strict mode with the IOB2 format. This
metric provides an entity-level evaluation based on the first token of each
entity, instead of a token-level basis. This approach offers a tokenizer-
agnostic assessment, reducing any potential correlation between the models’
performance and the tokenization process.

#### Text classification

For the document classification tasks, a sequence classification head is added
on top of the pooled output. Documents are truncated to the first 4,096 tokens
to match the length limits of the Longformer models. For the short-sequence
models, we truncate the document to the first 512 tokens. All classification
tasks are evaluated with a weighted F1 score.

#### Semantic textual similarity

The CLISTER and DEFT-2020 datasets are fine-tuned as sequence classification
with input formatted as [CLS] <text1> [SEP] <text2> [EOS]. No sequence
truncation is needed, since the maximum sequence lengths of these datasets are
less than 512 tokens. Predictions are made by adding a linear layer on top of
the pooled output. The evaluation metrics used are the mean relative solution
distance accuracy (EDRM), as defined by the original authors of the dataset
Cardon et al. (2020), and the Spearman correlation.

#### Multiple-choice question answering

The FrenchMedMCQA dataset is fine-tuned as sequence classification by adding a
linear layer on top of the pooled output. The input sequence is formatted as
follows: [CLS] <Question> [SEP] <Answer A> [SEP] <Answer B> [SEP] <Answer C>
[SEP] <Answer D> [SEP] <Answer E> [EOS]. As the maximum sequence lengths are
less than 512 tokens, no truncation is needed. For the output, the
classification is multi-class with 31 classes corresponding to existing
combinations of answers in the corpus. For example, if the correct answers are
A and B, we consider the correct class being AB. The models’ performance is
assessed using exact match ratio (EMR) and Hamming score.

Dataset | Task | DrBERT | CamemBERT-bio | Clinical-Longformer | DrBERT-4096 | DrLongformer-CP | DrLongformer-FS
---|---|---|---|---|---|---|---
ESSAI | POS | 98.41* | 97.39* | 98.54* |  98.60* | 98.72 | 98.48*
CAS | POS | 96.93* | 95.22* | 97.26* |  97.55* | 97.73 | 97.29*
CLISTER | STS | 0.62 / 0.57* | 0.54* / 0.26* | 0.6 / 0.88 |  0.65 / 0.71 | 0.6 / 0.86 |  0.64 / 0.58*
DEFT-2020 | STS | 0.72 / 0.81* | 0.58* / 0.32* | 0.71 / 0.85 |  0.73 / 0.84 |  0.78 / 0.88 | 0.65 / 0.79*
CLS | 82.38 | 94.78 | 95.59 | 87.48 | 94.27 | 36.57*
DEFT-2021 | CLS | 34.15* | 17.82* | 27.73* | 36.98* | 51.23 | 49.54
NER | 60.44* | 64.36 | 55.62* | 59.74* |  63.73* | 55.00*
QUAERO-EMEA | NER | 64.11 | 66.59 | 50.36* | 64.09 | 64.78 | 66.31
E3C | NER-Clinical | 54.45 | 56.96 | 50.88* | 57.20 | 57.07 | 55.74
NER-Temporal | 81.48* | 83.44 | 79.11* | 81.98* | 83.32 | 79.23*
FrenchMedMCQA | MCQA |  31.07* / 3.22* |  35.3 / 1.45* | 28.55* / 1.45* | 28.2* / 3.54 | 28.62* / 1.45* | 23.5* / 3.38
CLS | 65.38 | 65.79 | 64.57 | 66.59 | 66.0 | 64.42
MorFITT | CLS | 68.70* | 67.53* | 62.98* |  70.75* | 71.95 | 69.94*
DiaMed | CLS | 60.45 | 39.57* | 39.91* | 61.17 | 54.82* | 54.05*
aHF classification | CLS | 83.25* | 83.47* | 79.14* |  85.22* | 87.42 | 82.34*
aHF NER | NER | 37.32 | 29.08* | 30.86* | 38.53 | 36.44* | 34.81*

Table 3: Performance of the pretrained models on French biomedical and
clinical downstream tasks. Best model in bold and second is underlined.
Statistical significance is computed for each task using Student’s t-test with
the best model as reference: * stands for p < 0.05.

##  5 Results and Discussions

The performance of the models on downstream tasks is presented in Table 3. We
propose to analyze the results from two perspectives: (i) pre-training
strategies and (ii) model architecture, considering both the maximum input
sequence length and the attention mechanism.

###  5.1 Impact of the pre-training strategies

#### DrBERT vs CamemBERT-bio

When comparing models solely based on the BERT architecture, we find that
DrBERT and CamemBERT-bio have comparable performances. Specifically, each
model outperforms in 8 of the 16 tasks evaluated. As described in Table 4,
CamemBERT-bio shows a slight advantage in NER tasks, with an average F1 score
difference of 0.53. Conversely, DrBERT exhibits superior performance in
classification tasks, yielding an average F1 score difference of 4.23. These
observations indicate that there are ultimately no real differences on
downstream tasks between training from scratch and continual pre-training for
BERT models.

From an environmental impact perspective, continual pre-training appears to be
the most responsible approach. In fact, the estimated carbon emission for pre-
training CamemBERT-bio is 0.84 kg CO2eq. This estimation doesn’t take into
account the carbon emissions of the CamemBERT model itself. In contrast, the
DrBERT model has emissions amounting to 36.9 kg CO2eq, which is approximately
44 times higher. Given that specialized domain models are built on existing
architectures, it is more advantageous to continue pre-training from these
models.

#### Pre-training strategies for long-sequence models

Contrary to what we observed with BERT biomedical models, using an existing
model for pre-training—whether through conversion of BERT or continual pre-
training of Longformer—yields better results than training a model from
scratch. Indeed, the DrLongformer-FS model never outperforms either
DrBERT-4096 or DrLongformer-CP. The results also highlights that DrLongformer-
FS is better than Clinical-Longformer on only 10 of the 16 tasks. However,
it’s important to note that none of the Longformer models mentioned in the
literature have been pretrained from scratch. For example, Clinical-Longformer
was pretrained from the weights of Longformer, which in turn was pretrained
from the weights of RoBERTa. In the end, the DrLongformer-CP model underwent
four pre-training phases, resulting in enhanced robustness due to the variety
of data seen during the multiple pre-training phases. For a fair comparison
between DrLongformer-FS and other models, DrLongformer-FS would need a pre-
training duration equivalent to the multi-phase pre-training experienced by
the other models. However, this approach would be prohibitively costly, with
no guarantee of significant performance improvement.

When comparing DrBERT-4096 and DrLongformer-CP, the results are more nuanced.
DrLongformer-CP, which was pretrained from the weights of Clinical-Longformer,
achieves better results in 11 of the 16 tasks. The DrLongformer-CP model
appears to benefit from the clinical data from the MIMIC database used to pre-
train the Clinical-Longformer model. This finding supports the conclusions
drawn by Labrak et al. (2023a) regarding the effectiveness of specialized
knowledge transfer between languages, particularly when further pretraining an
English clinical model on French biomedical data. However, it remains
uncertain whether the continual pretraining from a specialized English model
offers more advantages than from a general French model, given the absence of
a French Longformer model for such an experiment. While the DrBERT-4096 model
doesn’t perform as well as the DrLongformer-CP, it’s important to note that it
offers a more cost-effective option for adapting a new architecture to a
specialized domain. Despite its lower training cost, DrBERT-4096 still
delivers performance levels that are comparably close to DrLongformer-CP
across many tasks.

###  5.2 Impact of the model architecture on downstream tasks

Regarding the architectures of pretrained models, the results show that
Longformer models perform better than BERT models on 11 of the 16 tasks. By
analyzing the results according to the type of evaluation task from Table 3,
we can draw different conclusions.

Task | DrBERT | CamemBERT-bio | Clinical-Longformer | DrBERT-4096 | DrLongformer-CP | DrLongformer-FS
---|---|---|---|---|---|---
POS | 97.67 | 96.31 | 97.90 | 98.08 | 98.23 | 97,89
CLS | 65.72 | 61.49 | 61.65 | 68.03 | 70.95 | 59.48
NER | 59.56 | 60.09 | 53.37 | 60.31 | 61.07 | 58.22
STS |  0.67 / 0.69 | 0.56 / 0.29 | 0.66 / 0.87 |  0.69 / 0.78 |  0.69 / 0.87 | 0.65 / 0.69
MCQA |  31.07 / 3.22 |  35.3 / 1.45 | 28.55 / 1.45 | 28.2 / 3.54 | 28.62 / 1.45 | 23.5 / 3.38

Table 4: Average performance per task. Best model in bold and second is underlined. | DEFT-2021 | DiaMed | MorFITT | aHF classification
---|---|---|---|---
Sequence length |  $\leq$ 388 |  $>$ 388 |  $\leq$ 395 |  $>$ 395 |  $\leq$ 406 |  $>$ 406 |  $\leq$ 400 |  $>$ 400
Documents in test set | 78 | 30 | 112 | 42 | 1056 | 32 | 54 | 274
DrBERT | 16.19 | 27.52 | 52.22 | 47.37 | 8.19 | 9.32 | 5.64 | 14.70
DrBERT-4096 | 17.31 | 28.10 | 48.56 | 36.27 | 8.94 | 9.57 | 9.68 | 13.87
DrLongformer-CP | 14.31 | 22.09 | 58.53 | 38.73 | 7.94 | 10.16 | 4.03 | 7.55
Table 5: Error rates ($\downarrow$) of DrBERT, DrBERT-4096 and DrLongformer-CP
on the test set of DEFT-2021 and aHF classification datasets. The selected
word sequence length threshold for each dataset corresponds to the maximum
length limit for DrBERT. Best model in bold.

(a) aHF classification - Binary classification.

(b) DEFT-2021 - Multi-label classification.

Figure 1: Average attention weights for each word position on the test set of
(a) the aHF classification dataset, and (b) the DEFT-2021 dataset. The
attention weights are obtained from the [CLS] token used for classification by
summing the attention weights of all attention heads in the last layer of the
model. These weights represent the most important words after they have
already incorporated contextual information from other words based off the 12
self-attention layers of the model. In (a) The aHF classification task is
annotated at both document-level (Yes/No) and sequence-level. At the sequence-
level, specific sequences of words that justify the document-level
classification are annotated. This enables a comparison between human
annotations and attention mechanisms of the models.

#### Named entity recognition

We notice that the BERT models are more suited for NER tasks. Indeed, even
though DrBERT and CamemBERT-bio process NER tasks at the sentence level rather
than the document level, they outperform the Longformer models in 3 of the 5
NER tasks. This shows that global context is not necessarily required for NER
tasks, and that the local context is generally sufficient for the evaluated
datasets. Although BERT models exhibit superior performance on individual NER
tasks, Table 4 shows that DrLongformer-CP achieves a higher average F1 score
across all tasks, demonstrating the model’s robustness.

#### Document classification

Sequence classification tasks appear to benefit from the Longformer’s maximum
sequence length of 4,096 tokens. For tasks that involve sequences longer than
512 tokens, such as those in DEFT-2021, MorFITT, DiaMed, or aHF
classification, the results show a clear advantage for Longformer models. On
average, there is a gain of 6.25 points in the F1 score when comparing the
best BERT model to the best Longformer model. For these four datasets, we
assessed the error rates for both short and long sequences. This analysis
aimed to determine on which sequence type the Longformer models demonstrated
improved performance. In Table 5, the error rates indicate that Longformer
models outperform DrBERT on both short and long sequences. For short
sequences, the average difference in error rates between DrBERT and the best-
performing Longformer model is 1.85%. For long sequences, this difference
increases to 5.86%. Moreover, in tasks with shorter sequences, such as
FrenchMedMCQA and DEFT-2020, the Longformer models still show a slight edge,
with an average F1 score improvement of 0.81 over the BERT variants. This
demonstrates a consistent advantage of Longformer models across a range of
sequence lengths and classification tasks.

By analyzing the distribution of attention on the clinical reports from the
aHF classification dataset in Figure 1a, we observe that the attention is
focused at the beginning of the document, in the same place as human
annotations. Indeed, French clinical notes always start with the reason for
the hospital stay, which is an important marker for the classification. This
concentration of attention at the beginning of the documents also explains why
BERT achieves good performance on this task, despite truncation to the first
512 tokens. The attention peaks after the 1,500 word position correspond to
the conclusions found at the end of clinical notes, which summarize the
important elements of the clinical case. It is also important to note that the
spikes are more prominent at the tail of the figure because the average is
calculated over a smaller number of documents. On the DEFT-2021 dataset, the
model attention is evenly distributed across the entire document, as
illustrated in Figure 1b. These two examples highlight the necessity of
tailoring models to handle documents of different lengths and ensure that
essential information is not missed.

#### Multiple-choice question answering

On the multiple-choice question answering task from the FrenchMedMCQA corpus,
BERT models appear to be more efficient with EMR scores exceeding 30% compared
to 28.62% for the best Longformer models. In Table 3, the two metrics often
show opposite results. DrBERT-4096 and DrLongformer-FS exhibit EMR scores
exceeding 3.0 but lower Hamming scores (below 30%). In such instances, these
models perform well in answering questions with a single correct answer but
struggle when confronted with questions that have multiple answer variations
to consider. On the contrary, while CamemBERT-bio excels at providing partial
answers to questions, achieving a Hamming score of 35.3%, it encounters more
challenges when tasked with questions that have a single, definitive answer.

#### Semantic textual similarity

The reported results between the models are very close on DEFT-2020 and
CLISTER datasets, with neither BERT nor Longformer distinctly standing out. In
terms of Spearman correlation, there is no significant difference between BERT
and Longformer.

##  6 Conclusion

In this work, we proposed a comparative study of adaptation strategies for
French biomedical and clinical long-sequence models. Specifically, we have
shown that there is no significant difference in performances between pre-
training from scratch and continual pre-training for models based on BERT
architecture. On the other hand, for larger models, converting or further pre-
train an existing model yields better performance than training from scratch.
We have also demonstrated for the continual pre-training strategy that
building a specialized French model from a specialized English model is
possible. Using an English specialized model becomes a good alternative when
there is no French-specific model available for the desired architecture.
These findings suggest that domain transfer can be achieved cross-lingually
through multi-phase adaptive pre-training. Furthermore, the lack of open
French clinical data can be compensated by combining English clinical data
from the MIMIC database with French biomedical data.

## Limitations

The comparative study conducted here is difficult to reproduce, since it
required a considerable amount of computational resources. Approximately 5,000
hours of GPU were necessary to pre-train the three new models introduced in
this study. For the evaluation of these models across the evaluation tasks,
1,536 fine-tuning sessions were conducted, for a total of 4,400 GPU hours. The
fine-tuning cost is higher for Longformer models than for BERT models. As the
model size increases, there’s a greater need to reduce the batch size. In
total, approximately 9,400 hours of GPU computation were used for this work.
The environmental cost is equivalent to 2,434.6 KWh or 138.77 kg CO2eq.

## Acknowledgements

This work was performed using HPC resources from GENCI-IDRIS (Grant
2023-AD011013715R1) and CCIPL (Centre de Calcul Intensif des Pays de la
Loire). This work was financially supported by ANR AIBy4 (ANR-20-THIA-0011).

## References

  * Beltagy et al. (2019) Iz Beltagy, Kyle Lo, and Arman Cohan. 2019.  Scibert: A pretrained language model for scientific text.  _arXiv preprint arXiv:1903.10676_.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv:2004.05150_.
  * Berhe et al. (2023) Aman Berhe, Guillaume Draznieks, Vincent Martenot, Valentin Masdeu, Lucas Davy, and Jean-Daniel Zucker. 2023.  AliBERT: A pre-trained language model for French biomedical text.  In _The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks_ , pages 223–236, Toronto, Canada. Association for Computational Linguistics.
  * Bressem et al. (2023) Keno K. Bressem, Jens-Michalis Papaioannou, Paul Grundmann, Florian Borchert, Lisa C. Adams, Leonhard Liu, Felix Busch, Lina Xu, Jan P. Loyen, Stefan M. Niehues, Moritz Augustin, Lennart Grosser, Marcus R. Makowski, Hugo JWL. Aerts, and Alexander Löser. 2023.  Medbert.de: A comprehensive german bert model for the medical domain.  _arXiv preprint arXiv:2303.08179_.  Keno K. Bressem and Jens-Michalis Papaioannou and Paul Grundmann contributed equally.
  * Cardon et al. (2020) Rémi Cardon, Natalia Grabar, Cyril Grouin, and Thierry Hamon. 2020.  Présentation de la campagne d’évaluation DEFT 2020 : similarité textuelle en domaine ouvert et extraction d’information précise dans des cas cliniques (presentation of the DEFT 2020 challenge : open domain textual similarity and precise information extraction from clinical cases ).  In _Actes de la 6e conférence conjointe Journées d’Études sur la Parole (JEP, 33e édition), Traitement Automatique des Langues Naturelles (TALN, 27e édition), Rencontre des Étudiants Chercheurs en Informatique pour le Traitement Automatique des Langues (RÉCITAL, 22e édition). Atelier DÉfi Fouille de Textes_ , pages 1–13, Nancy, France. ATALA et AFCP.
  * Carrino et al. (2021) Casimiro Pio Carrino, Jordi Armengol-Estapé, Asier Gutiérrez-Fandiño, Joan Llop-Palao, Marc Pàmies, Aitor Gonzalez-Agirre, and Marta Villegas. 2021.  Biomedical and clinical language models for spanish: On the benefits of domain-specific pretraining in a mid-resource scenario.  _arXiv preprint arXiv:2109.03570_.
  * Choromanski et al. (2020a) Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, David Belanger, Lucy Colwell, et al. 2020a.  Masked language modeling for proteins via linearly scalable long-context transformers.  _arXiv preprint arXiv:2006.03555_.
  * Choromanski et al. (2020b) Krzysztof Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. 2020b.  Rethinking attention with performers.  _arXiv preprint arXiv:2009.14794_.
  * Condevaux and Harispe (2023) Charles Condevaux and Sébastien Harispe. 2023.  Lsg attention: Extrapolation of pretrained transformers to long sequences.  In _Advances in Knowledge Discovery and Data Mining_ , pages 443–454, Cham. Springer Nature Switzerland.
  * Dalloux et al. (2021) Clément Dalloux, Vincent Claveau, Natalia Grabar, Lucas Emanuel Silva Oliveira, Claudia Maria Cabral Moro, Yohan Bonescki Gumiel, and Deborah Ribeiro Carvalho. 2021.  Supervised learning for the detection of negation and of its scope in French and Brazilian Portuguese biomedical corpora.  _Natural Language Engineering_ , 27(2):181–201.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Fries et al. (2022) Jason Alan Fries, Leon Weber, Natasha Seelam, Gabriel Altay, Debajyoti Datta, Samuele Garda, Myungsun Kang, Ruisi Su, Wojciech Kusa, Samuel Cahyawijaya, Fabio Barth, Simon Ott, Matthias Samwald, Stephen Bach, Stella Biderman, Mario Sänger, Bo Wang, Alison Callahan, Daniel León Periñán, Théo Gigant, Patrick Haller, Jenny Chim, Jose David Posada, John Michael Giorgi, Karthik Rangasai Sivaraman, Marc Pàmies, Marianna Nezhurina, Robert Martin, Michael Cullan, Moritz Freidank, Nathan Dahlberg, Shubhanshu Mishra, Shamik Bose, Nicholas Michio Broad, Yanis Labrak, Shlok S Deshmukh, Sid Kiblawi, Ayush Singh, Minh Chien Vu, Trishala Neeraj, Jonas Golde, Albert Villanova del Moral, and Benjamin Beilharz. 2022.  Bigbio: A framework for data-centric biomedical natural language processing.  In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
  * Gao et al. (2021) Shang Gao, Mohammed Alawad, M Todd Young, John Gounley, Noah Schaefferkoetter, Hong Jun Yoon, Xiao-Cheng Wu, Eric B Durbin, Jennifer Doherty, Antoinette Stroup, et al. 2021.  Limitations of transformers on clinical text classification.  _IEEE journal of biomedical and health informatics_ , 25(9):3596–3607.
  * Grabar et al. (2018) Natalia Grabar, Vincent Claveau, and Clément Dalloux. 2018.  CAS: French Corpus with Clinical Cases.  In _Proceedings of the 9th International Workshop on Health Text Mining and Information Analysis (LOUHI)_ , pages 1–7, Brussels, Belgium.
  * Grouin et al. (2021) Cyril Grouin, Natalia Grabar, and Gabriel Illouz. 2021.  Classification de cas cliniques et évaluation automatique de réponses d’étudiants : présentation de la campagne DEFT 2021 (clinical cases classification and automatic evaluation of student answers : Presentation of the DEFT 2021 challenge).  In _Actes de la 28e Conférence sur le Traitement Automatique des Langues Naturelles. Atelier DÉfi Fouille de Textes (DEFT)_ , pages 1–13, Lille, France. ATALA.
  * Gu et al. (2021) Yu Gu, Robert Tinn, Hao Cheng, Michael Lucas, Naoto Usuyama, Xiaodong Liu, Tristan Naumann, Jianfeng Gao, and Hoifung Poon. 2021.  Domain-specific language model pretraining for biomedical natural language processing.  _ACM Trans. Comput. Healthcare_ , 3(1).
  * Han et al. (2023) Tianyu Han, Lisa C Adams, Jens-Michalis Papaioannou, Paul Grundmann, Tom Oberhauser, Alexander Löser, Daniel Truhn, and Keno K Bressem. 2023.  Medalpaca–an open-source collection of medical conversational ai models and training data.  _arXiv preprint arXiv:2304.08247_.
  * He et al. (2023) Kai He, Rui Mao, Qika Lin, Yucheng Ruan, Xiang Lan, Mengling Feng, and Erik Cambria. 2023.  A survey of large language models for healthcare: from data, technology, and applications to accountability and ethics.
  * Hiebel et al. (2022) Nicolas Hiebel, Olivier Ferret, Karën Fort, and Aurélie Névéol. 2022.  CLISTER : A corpus for semantic textual similarity in French clinical narratives.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 4306–4315, Marseille, France. European Language Resources Association.
  * Huang et al. (2019) Kexin Huang, Jaan Altosaar, and Rajesh Ranganath. 2019.  Clinicalbert: Modeling clinical notes and predicting hospital readmission.  _arXiv preprint arXiv:1904.05342_.
  * Johnson et al. (2016) Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. 2016.  Mimic-iii, a freely accessible critical care database.  _Scientific data_ , 3(1):1–9.
  * Labrak et al. (2022) Yanis Labrak, Adrien Bazoge, Richard Dufour, Béatrice Daille, Pierre-Antoine Gourraud, Emmanuel Morin, and Mickael Rouvier. 2022.  FrenchMedMCQA: A French Multiple-Choice Question Answering Dataset for Medical domain.  In _Proceedings of the 13th International Workshop on Health Text Mining and Information Analysis (LOUHI)_ , Abou Dhabi, United Arab Emirates.
  * Labrak et al. (2023a) Yanis Labrak, Adrien Bazoge, Richard Dufour, Mickael Rouvier, Emmanuel Morin, Béatrice Daille, and Pierre-Antoine Gourraud. 2023a.  DrBERT: A robust pre-trained model in French for biomedical and clinical domains.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 16207–16221, Toronto, Canada. Association for Computational Linguistics.
  * Labrak et al. (2024) Yanis Labrak, Adrien Bazoge, Oumaima El Khettari, Mickael Rouvier, Pacome Constant dit Beaufils, Natalia Grabar, Beatrice Daille, Solen Quiniou, Emmanuel Morin, Pierre-Antoine Gourraud, and Richard Dufour. 2024.  Drbenchmark: A large language understanding evaluation benchmark for french biomedical domain.  _arXiv preprint arXiv:2402.13432_.
  * Labrak et al. (2023b) Yanis Labrak, Mickaël Rouvier, and Richard Dufour. 2023b.  MORFITT : A multi-label corpus of French scientific articles in the biomedical domain.  In _30e Conférence sur le Traitement Automatique des Langues Naturelles (TALN) Atelier sur l’Analyse et la Recherche de Textes Scientifiques_ , Paris, France. Florian Boudin.
  * Labrak et al. (2023c) Yanis Labrak, Mickael Rouvier, and Richard Dufour. 2023c.  A zero-shot and few-shot study of instruction-finetuned large language models applied to clinical and biomedical tasks.  _arXiv preprint arXiv:2307.12114_.
  * Lee et al. (2019) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019.  BioBERT: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240.
  * Li et al. (2023) Yikuan Li, Ramsey M Wehbe, Faraz S Ahmad, Hanyin Wang, and Yuan Luo. 2023.  A comparative study of pretrained language models for long clinical text.  _Journal of the American Medical Informatics Association_ , 30(2):340–347.
  * Lindberg et al. (1993) DA Lindberg, BL Humphreys, and AT McCray. 1993.  The Unified Medical Language System.  _Methods Inf Med_ , 32(4):281–291.
  * Ma et al. (2021) Xuezhe Ma, Xiang Kong, Sinong Wang, Chunting Zhou, Jonathan May, Hao Ma, and Luke Zettlemoyer. 2021.  Luna: Linear unified nested attention.  _Advances in Neural Information Processing Systems_ , 34:2441–2453.
  * Magnini et al. (2020) Bernardo Magnini, Begoña Altuna, Alberto Lavelli, Manuela Speranza, and Roberto Zanoli. 2020.  The e3c project: Collection and annotation of a multilingual corpus of clinical cases.  _Proceedings of the Seventh Italian Conference on Computational Linguistics CLiC-it 2020_.
  * Martin et al. (2020) Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric de la Clergerie, Djamé Seddah, and Benoît Sagot. 2020.  CamemBERT: a tasty French language model.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7203–7219, Online. Association for Computational Linguistics.
  * Micikevicius et al. (2017) Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. 2017.  Mixed precision training.  _CoRR_ , abs/1710.03740.
  * Nakayama (2018) Hiroki Nakayama. 2018.  seqeval: A python framework for sequence labeling evaluation.  Software available from https://github.com/chakki-works/seqeval.
  * Névéol et al. (2014) Aurélie Névéol, Cyril Grouin, Jeremy Leixa, Sophie Rosset, and Pierre Zweigenbaum. 2014.  The QUAERO French medical corpus: A ressource for medical entity recognition and normalization.  In _Proc of BioTextMining Work_ , pages 24–30.
  * Peng et al. (2021) Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. 2021.  Random feature attention.  In _International Conference on Learning Representations_.
  * Peng et al. (2019) Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019.  Transfer learning in biomedical natural language processing: An evaluation of BERT and ELMo on ten benchmarking datasets.  In _Proceedings of the 18th BioNLP Workshop and Shared Task_ , pages 58–65, Florence, Italy. Association for Computational Linguistics.
  * Ramachandran et al. (2023) Giridhar Kaushik Ramachandran, Yujuan Fu, Bin Han, Kevin Lybarger, Nic Dobbins, Ozlem Uzuner, and Meliha Yetisgen. 2023.  Prompt-based extraction of social determinants of health using few-shot learning.  In _Proceedings of the 5th Clinical Natural Language Processing Workshop_ , pages 385–393, Toronto, Canada. Association for Computational Linguistics.
  * Schneider et al. (2020) Elisa Terumi Rubel Schneider, João Vitor Andrioli de Souza, Julien Knafou, Lucas Emanuel Silva e Oliveira, Jenny Copara, Yohan Bonescki Gumiel, Lucas Ferro Antunes de Oliveira, Emerson Cabrera Paraiso, Douglas Teodoro, and Cláudia Maria Cabral Moro Barra. 2020.  BioBERTpt - a Portuguese neural language model for clinical named entity recognition.  In _Proceedings of the 3rd Clinical Natural Language Processing Workshop_ , pages 65–72, Online. Association for Computational Linguistics.
  * Singhal et al. (2023) Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Le Hou, Kevin Clark, Stephen Pfohl, Heather Cole-Lewis, Darlene Neal, et al. 2023.  Towards expert-level medical question answering with large language models.  _arXiv preprint arXiv:2305.09617_.
  * Tay et al. (2020) Yi Tay, Dara Bahri, Liu Yang, Donald Metzler, and Da-Cheng Juan. 2020.  Sparse sinkhorn attention.  In _International Conference on Machine Learning_ , pages 9438–9447. PMLR.
  * Tay et al. (2021) Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. 2021.  Long range arena : A benchmark for efficient transformers.  In _International Conference on Learning Representations_.
  * Tay et al. (2022) Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. 2022.  Efficient transformers: A survey.  _ACM Comput. Surv._ , 55(6).
  * Touchent et al. (2023) Rian Touchent, Laurent Romary, and Eric de la Clergerie. 2023.  Camembert-bio: a tasty french language model better for your health.
  * Turkmen et al. (2022) Hazal Turkmen, Oguz Dikenelli, Cenk Eraslan, and Mehmet Cem Callı. 2022.  Bioberturk: Exploring turkish biomedical language model development strategies in low resource setting.  _Preprint from Research Square_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Wang et al. (2020) Sinong Wang, Belinda Z Li, Madian Khabsa, Han Fang, and Hao Ma. 2020.  Linformer: Self-attention with linear complexity.  _arXiv preprint arXiv:2006.04768_.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.
  * Wu et al. (2023) Chaoyi Wu, Xiaoman Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. 2023.  Pmc-llama: Further finetuning llama on medical papers.  _arXiv preprint arXiv:2304.14454_.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Łukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016.  Google’s neural machine translation system: Bridging the gap between human and machine translation.
  * Yuan et al. (2023) Chenhan Yuan, Qianqian Xie, and Sophia Ananiadou. 2023.  Zero-shot temporal relation extraction with ChatGPT.  In _The 22nd Workshop on Biomedical Natural Language Processing and BioNLP Shared Tasks_ , pages 92–102, Toronto, Canada. Association for Computational Linguistics.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020.  Big bird: Transformers for longer sequences.  _Advances in neural information processing systems_ , 33:17283–17297.
  * Zhou et al. (2023) Weipeng Zhou, Majid Afshar, Dmitriy Dligach, Yanjun Gao, and Timothy Miller. 2023.  Improving the transferability of clinical note section classification models with BERT and large language model ensembles.  In _Proceedings of the 5th Clinical Natural Language Processing Workshop_ , pages 125–130, Toronto, Canada. Association for Computational Linguistics.


# Pair density wave order from electron repulsion

Yi-Ming Wu  Stanford Institute for Theoretical Physics, Stanford University,
Stanford, California 94305, USA  P. A. Nosov  Stanford Institute for
Theoretical Physics, Stanford University, Stanford, California 94305, USA
Aavishkar A. Patel  Center for Computational Quantum Physics, Flatiron
Institute, New York NY 10010, USA  S. Raghu  Stanford Institute for
Theoretical Physics, Stanford University, Stanford, California 94305, USA

###### Abstract

A pair density wave (PDW) is a superconductor whose order parameter is a
periodic function of space, without an accompanying spatially-uniform
component. Since PDWs are not the outcome of a weak-coupling instability of a
Fermi liquid, a generic pairing mechanism for PDW order has remained elusive.
We describe and solve models having robust PDW phases. To access the
intermediate coupling limit, we invoke large $N$ limits of Fermi liquids with
repulsive BCS interactions that admit saddle point solutions. We show that the
requirements for long range PDW order are that the repulsive BCS couplings
must be non-monotonic in space and that their strength must exceed a threshold
value. We obtain a phase diagram with both finite temperature transitions to
PDW order, and a $T=0$ quantum critical point, where non-Fermi liquid behavior
occurs.

Introduction.  A pair density wave (PDW) is a rare and exotic superconductor
in which pairs of electrons condense with non-zero center of mass momentum[1].
Similar phases of matter were conceived decades ago by Fulde, Ferrel, Larkin
and Ovchinnikov (FFLO), in the context of spin-polarized superconductivity[2,
3, 4, 5, 6, 7]. In addition to exhibiting the usual properties of
superconductors, PDWs break translation symmetry and are therefore accompanied
by charge modulation. PDW order is believed to occur in a variety of
correlated electron materials[8, 9, 10, 11, 12, 13, 14, 15, 16, 17], in cold
atom systems[18, 19, 20], and in some systems with nested Fermi surfaces[21,
22, 23, 24, 25, 26]. More recently, they have been observed in the Iron based
superconductor EuRbFe4As4[27] as well as the Kagome metal CsV3Sb5[28]. Since
PDWs do not stem from a weak-coupling instability of a Fermi liquid, robust
mechanisms of PDW formation have remained elusive, despite intense efforts[29,
30, 31, 32, 33, 34, 14, 35] .

It is easy to see why PDW order requires intermediate coupling. In a clean
Fermi liquid with inversion and/or time-reversal symmetry, the static pair
susceptibility is a positive-definite quantity that diverges logarithmically
only at zero center of mass momentum $\bm{q}=0$, reflecting the celebrated BCS
instability. Away from $\bm{q}=0$, the logarithmic divergence is cut off, and
pairing with $\bm{q}\neq 0$ requires a finite interaction strength. Therefore,
many proposed mechanisms for FFLO superconductivity have relied on shifting
the large pair susceptibility away from $\bm{q}=0$, say by the application of
a zeeman field[2, 3], or, say, by considering the effects of Rashba spin-orbit
effects on odd parity superconductivity[36]. By contrast, we wish to ask
whether there can be an intrinsic mechanism for PDW order, which does not rely
on a proximity to $\bm{q}=0$ pairing, instead, requiring only the existence of
sizeable interactions.

Figure 1: Phase diagram obtained from the large-$N$ model. At $T=0$, there is
a quantum critical point (QCP) separating the PDW phase and the normal
metallic state. The PDW transition temperature $T_{c}$ scales linear in
$g^{2}$ in strong coupling limit. Above the QCP, fluctuation of PDW gives rise
to NFL behavior.

In this letter, we study various models of Fermi liquids in the presence of
repulsive BCS interactions. We solve such theories beyond the weak-coupling
regime by appealing to a large $N$ limit whose saddle point corresponds to a
self-consistent set of solutions for the propagators of the theory. From these
solutions, we deduce the existence of both finite temperature continuous
transitions to PDW order, as well as a quantum critical point at $T=0$
separating a Fermi liquid metal from a PDW. Our analysis leads to robust
pairing mechanisms in $d>1$ of PDW order in a variety of continuum and lattice
systems. Despite such robustness, we find that PDW order emerges from
physically reasonable microscopic models only under special circumstances,
which we precisely outline below. This perhaps accounts in part for why PDW
order is so rare in real materials.

Model and method of solution.  We will study the fate of a Fermi liquid
subject to a finite repulsive singlet BCS interaction:

$H_{\text{pair}}=\sum_{ij}V_{ij}b_{i}^{\dagger}b_{j},~{}b_{i}=c_{i\downarrow}c_{i\uparrow},$
(1)

In a translationally invariant system, $V_{ij}=V(\bm{r}_{i}-\bm{r}_{j})$, and
the interaction above can equivalently be expressed in momentum space as
$H_{\text{pair}}=\sum_{\bm{q}}V(\bm{q})b^{\dagger}_{\bm{q}}b_{\bm{q}}$. We
decouple the above interaction using an auxiliary field $\phi$, which
corresponds to a charge $2e$ pair field. The bare euclidean Lagrangian density
then consists of the metal, the pair fields, and a Yukawa coupling between
them: $\mathcal{L}=\mathcal{L}_{f}+\mathcal{L}_{b}+\mathcal{L}_{g}$, where

$\displaystyle\mathcal{L}_{f}$ $\displaystyle=$
$\displaystyle\sum_{\sigma=\uparrow,\downarrow}\int_{y}\psi_{\sigma}^{\dagger}(x)G_{0}^{-1}(x-y)\psi_{\sigma}(y),$
$\displaystyle\mathcal{L}_{b}$ $\displaystyle=$
$\displaystyle\int_{y}\phi^{\dagger}(x)D_{0}^{-1}(x-y)\phi(y),$
$\displaystyle\mathcal{L}_{g}$ $\displaystyle=$ $\displaystyle\eta
g\left(\phi^{\dagger}(x)\psi_{\uparrow}(x)\psi_{\downarrow}(x)+\phi(x)\psi_{\downarrow}^{\dagger}(x)\psi_{\uparrow}^{\dagger}(x)\right),$
(2)

$x=\left(\bm{x},\tau\right)$, $\eta=1(i)$ corresponds to attractive
(repulsive) BCS couplings parametrized by a dimensionless coupling $g$ (for
the repulsive case, see [37] for details), and $G_{0},D_{0}$ are respectively
the bare fermion and boson propagators in the decoupled limit $g=0$ (i.e.
$D_{0}$ is proportional to the Fourier transform of the inverse
$[V(\mathbf{q})]^{-1}$).

The theory above can be solved for arbitrary $g$ using saddle point methods.
To do this, we consider a formal extension to large $N$ limit where the
fermion and boson fields are promoted to $N$ component vectors that transform
in the fundamental representation of a global $SU(N)$ flavor symmetry group.
The coupling between the fields is promoted to an all-to-all random Yukawa
coupling in the space of flavors:

$\mathcal{L}_{g}\rightarrow\eta\sum_{{\color[rgb]{0,0,0}km\ell}}\left(\frac{g_{{\color[rgb]{0,0,0}km\ell}}}{N}\psi_{{\color[rgb]{0,0,0}k}\uparrow}(x)\psi_{{\color[rgb]{0,0,0}m}\downarrow}(x)\phi^{\dagger}_{{\color[rgb]{0,0,0}\ell}}(x)+\frac{g^{*}_{{\color[rgb]{0,0,0}km\ell}}}{N}\psi^{\dagger}_{{\color[rgb]{0,0,0}m}\uparrow}(x)\psi^{\dagger}_{{\color[rgb]{0,0,0}k}\downarrow}(x)\phi_{{\color[rgb]{0,0,0}\ell}}(x)\right),$
(3)

where the quenched random Yukawa couplings are spatially independent, and are
chosen from a Gaussian unitary ensemble with variance

${\color[rgb]{0,0,0}\overline{g_{km\ell}g^{*}_{k^{\prime}m^{\prime}\ell^{\prime}}}=g^{2}\delta_{kk^{\prime}}\delta_{mm^{\prime}}\delta_{\ell\ell^{\prime}}}.$
(4)

and with zero average. The global $SU(N)$ symmetry is thus only preserved on
average. In terms of the original fermionic operators, this extension
corresponds to the interaction of the form

$H_{\text{pair}}=\sum_{ij}V_{ij}\sum\limits_{\ell}b_{\ell i}^{\dagger}b_{\ell
j},\;\;b_{\ell
i}=\sum\limits_{km}\frac{g_{km\ell}}{N}c_{ki\downarrow}c_{mi\uparrow}.$ (5)

Using by now standard saddle point methods[38, 39, 40, 41, 42], the exact
solution of the large N theory consists of self-consistent propagators, $G,D$
with associated self-energies $\Sigma,\Pi$:

$\displaystyle\Sigma(k)$
$\displaystyle=-g^{2}\sum_{q}\operatorname{sgn}[V(\bm{q})]G(-k+q)D(q),$ (6)
$\displaystyle\Pi(q)$
$\displaystyle=-g^{2}\operatorname{sgn}[V(\bm{q})]\sum_{k}G(k)G(-k+q),$
$\displaystyle G(k)$
$\displaystyle=\left[G_{0}^{-1}(k)+\Sigma(k)\right]^{-1},~{}D(q)=\left[D_{0}^{-1}(q)-\Pi(q)\right]^{-1}.$

Here, $k=(\bm{k},i\omega_{n})$ and $q=(\bm{q},i\Omega_{m})$, where
$\omega_{n}(\Omega_{m})$ are fermion(boson) Matsubara frequencies. The sign
function $\operatorname{sgn}[V(\bm{q})]$ originates from the factor $\eta$
introduced in Eq.(2). From the exact propagators $G,D$, we extract all the
salient physics, to obtain the schematic phase diagram in Fig. 1. For
instance, to identify the finite temperature PDW transitions shown in Fig. 1,
we need only consider the static bosonic propagator $D(\bm{q})$, having
integrated out the fermion fields. The effective Ginzburg-Landau theory for
the fields $\phi$ will have a quadratic term whose coefficient is given by
$D^{-1}(\bm{q})$. To study the manner in which the order parameter grows below
the PDW transition, we again integrate out the fermions and study static
bosonic propagators, but now with the inclusion of non-linear effects stemming
from a non-zero vacuum expectation value of $\phi$. Finally, we will describe
the PDW quantum critical point, in which both fermion and boson fields must
remain at play, and arrive at self-consistent solutions for $G,D$ including
frequency and momentum dependence.

Fluctuating PDW order.

Figure 2: $D^{-1}(\bm{q})$ in the zero temperature limit obtained from Eq.(7).
Here we set $c^{2}/r=0.5$, $\nu/r=0.1$, and the momentum is measured in units
of $4\omega_{D}/v_{F}$.

We first show that when the interaction $V(r)$ is monotonic, e.g. $V(r)\sim
e^{-r/\xi}$, the PDW order is absent for any $g$. The Fourier transform
$V(\bm{q})$ defines the bare inverse boson propagator, which is purely static,
and takes an Ornstein-Zernike form: $D_{0}^{-1}(\bm{q})=r+c^{2}q^{2}$, with
$r>0$. To see why the theory fails to host long range PDW order, consider the
limit $q/2k_{F}\ll 1$, in which the saddle point solution for the exact static
propagator $D$ at $T=0$ can be analytically obtained:

$D^{-1}(\bm{q})=r+c^{2}q^{2}+g^{2}\nu\log{\left(\frac{4\omega_{D}}{v_{F}q}\right)},$
(7)

where the last term above is the contribution from the $q\ll k_{F}$ limit of
the static pair susceptibility, $\omega_{D}$ is a cutoff, and $\nu$ is the
density of states at the Fermi level. Even at $T=0$, $D^{-1}(\bm{q})$ remains
positive, indicating the absence of a phase transition. Nevertheless, the
minimum of $D^{-1}(\bm{q})$ is at non-zero
$|\bm{q}|=\sqrt{\frac{g^{2}\nu}{2c^{2}}}$, indicating softened pair
fluctuations at finite momentum. Figure 2 shows $D^{-1}(\bm{q})$ for various
strengths $g^{2}$. With increasing $g^{2}$, the theory is driven further away
from ordering, eventually having a correlation length short compared to the
wavelength of the putative PDW - thus, a failed PDW. We next show how long
ranged PDW order trivially results when the repulsive BCS couplings are non-
monotonic in space.

PDWs from non-monotonic BCS interactions.

Figure 3: (a) Plot of $V(\bm{q})$ as a function of $|\bm{q}|r_{0}$ with
$r_{0}=1$ from the toy model in Eq.(8). (b) $D^{-1}(\bm{q})$ at $T=0.05$ as a
function of $|\bm{q}|r_{0}$ (also with $r_{0}=1$) obtained by approximating
$\Pi(\bm{q})$ in Eq.(9) by its one-loop calculation. (c) Density plot of
$D^{-1}(\bm{q})$ as a function of $\bm{q}$ obtained by numerically solving the
full saddle point equations in (6) with $r_{0}=3$. The two panels show the
results for $T$ above $T_{c}$ and right at $T_{c}$, and the dashed circles
mark the minimum of $D^{-1}(\bm{q})$. (d) $T_{c}$ as a function of $g^{2}$. At
large $g^{2}$, our result indicate that $T_{c}$ scales linearly in $g^{2}$.
(e) The magnitude of $\Delta(\bm{q})$ below $T_{c}$ for a given $g^{2}=0.45$.
The energy scale here are measured in unit of the Fermi energy $E_{F}$.

As an illustrative example, consider the case where the BCS coupling is non-
zero only at a distance $r_{0}$:

$V(\bm{r})=g^{2}\delta(r-r_{0}),\ V(\bm{q})=2\pi r_{0}g^{2}J_{0}(qr_{0}),$ (8)

where $J_{0}(x)$ is the zeroth Bessel function. Although $V(\bm{r})$ is
repulsive, its Fourier transform $V(\bm{q})$ is an oscillatory function with
both repulsive and attractive components. It has several local minima and a
global maximum at $q=0$ [left panel of Fig. 3(a)]. The exact boson propagator
in this case is

$D^{-1}(\bm{q})=\frac{1}{2\pi r_{0}|J_{0}(qr_{0})|}+g^{2}{\rm
sgn}\left[V(\bm{q})\right]\Pi(\bm{q}).$ (9)

To make sense of the above equation, we can approximate the boson self energy
$\Pi(\bm{q})$ by the one-loop calculation $\Pi_{0}(\bm{q})$ which is obtained
using $G_{0}$. The result is shown in Fig.3(b). Clearly we see that when
$V(\bm{q})<0$, the associated Fourier components of $D^{-1}(\bm{q})$ get
smaller (i.e. closer to an ordering transition) as $g^{2}$ increases whereas
the repulsive components get larger. Nonetheless, the phase transition will
not occur unless $g^{2}$ exceeds a threshold value. In Fig.3(c) we present the
numerical results of $D^{-1}(\bm{q})$ by solving the full saddle point
equations (6) on a $32\times 32$ momentum mesh grid. The global minimum
(dashed circle) of $D^{-1}(\bm{q})$ indeed vanishes when $T$ approaches
$T_{c}$. Thus, there is a line of finite temperature phase transitions
$T_{c}(g^{2})$ as $g^{2}$ is varies, obtained by the condition
$D^{-1}(\bm{q})=0$. For $T>T_{c}$, the minimum value of $D^{-1}(\bm{q})$ forms
a ring as is expected from the toy model, but stays positive. Once $T$
approaches $T_{c}$, its minimum vanishes, indicating the PDW instability.
Similarly, if we fix $T$ instead and increase $g^{2}$, we can also see
$D^{-1}(\bm{q})$ vanishes at some finite $g^{2}$. In Fig.3(d) we present
$T_{c}$ and a function of $g^{2}$. At large $g^{2}$, our results clearly shows
a linear relation between $T_{c}$ and $g^{2}$. The line of finite temperature
transitions terminates at a $T=0$ phase transition at $g=g_{c}$. We will
return to the quantum phase transition below.

Below the ordering transition, we must solve the self-consistent equations
allowing for a non-zero expectation value
$\Delta(\bm{q})=\langle\phi(\bm{q})\rangle$. Details of solution of the
resulting non-linear gap equations are provided in [37]. Fig. 3(e) shows
$\Delta(\bm{q})$ as a function of $T$ below $T_{c}$. Within the accuracy of
the numerical solutions, the expectation value grows continuously indicating
that the finite temperature transitions are second order and are well-decribed
by mean-field theory. From the solution of the non-linear equations, we can
also determine the ordering wave-vector $\bm{Q}$ of the PDW by minimizing
$D^{-1}(\bm{q})$ with respect to momentum:

$\bm{Q}:\frac{d}{d\bm{q}}D^{-1}(\bm{q})|_{\bm{Q}}=0$ (10)

In the neighborhood of $\bm{Q}$, $D^{-1}(\bm{q})$ takes the form
$D^{-1}(\bm{q})=\gamma\left({\color[rgb]{0,0,0}|q|-Q}\right)^{2}$, where
$\gamma=\frac{1}{2}\frac{d^{2}}{dq^{2}}D^{-1}(\bm{q})|_{\bm{q}}$.

Lattice models with PDW order. Emboldened by the simplified model above, we
consider a somewhat more realistic example of electrons on a square lattice
with nearest neighbor hopping $t=1$, onsite Hubbard $U$, and second neighbor
pair-hopping $J$:

$\displaystyle H=$
$\displaystyle-t\sum_{\braket{i,j},\sigma}c_{i\sigma}^{\dagger}c_{j\sigma}+U\sum_{i}n_{i\uparrow}n_{i\downarrow}$
(11)
$\displaystyle+J\sum_{\braket{i,j}}c_{i\uparrow}^{\dagger}c_{i\downarrow}^{\dagger}c_{j\downarrow}c_{j\uparrow},$

where $i,j$ above label lattice sites. The model above can similarly be
N-enhanced and the resulting saddle point solutions can be solved mutatis
mutandis.

Figure 4: Numerical solution of $D^{-1}(\bm{q})$ from Eq.(11) obtained at a
fixed $T=0.05$ with $J=2U$ for different fillings. In the left panel, there
are $n=0.71$ electrons per site, and $D^{-1}(\bm{q})$ touches zero at
$g^{2}=0.75$. In the right panel, there are $n=1.17$ electrons per site, and
$g^{2}=1.8$ is the critical coupling where $D^{-1}(\bm{q})$ touches zero. The
black dots mark the positions of the ordering vector $\bm{Q}$ near
$(\pm\pi,\pm\pi)$, which leads to the PDW with checkerboard pattern in real
space.

In this case, the Fourier transform of the BCS interaction $V(\bm{q})$ is

$V(\bm{q})=U+2J(\cos q_{x}+\cos q_{y}),$ (12)

and $g^{2}=U/t$. As long as $U<4J$, $V(\bm{q})$ can be negative at some finite
$\bm{q}$. We solve Eq.(6) with the fermion dispersion replaced with
$\xi_{\bm{k}}=-2t(\cos k_{x}+\cos k_{y})-\mu$. The results are shown in Fig.4
In this case, we have four symmetry-related ordering vectors at
$(\pm\pi,\pm\pi)+\mathcal{O}(U/J)$, that depend on the strength of
interactions and the filling. In this sense, the pairing state from the
large-$N$ theory is different from the $\eta$-pairing state found in numerical
studies of one dimensional analogs of such models[43, 44, 45, 46, 29].

PDW quantum critical point. Both the lattice and continuum models above have
finite temperature continuous PDW transitions that terminate at a quantum
phase transition. We can again study the behavior at this $T=0$ transition by
solving the self-consistent set of equations in Eq. (6). But now, we cannot
integrate out the fermions and must include dynamical properties of both
fermion and boson propagators to study their feedback onto one another. A
straightforward computation of the one-loop boson self-energy in the regime
$q\ll k_{F}$ yields (see supplementary section [37])
$\Pi(\bm{q},i\Omega_{m})=\nu(\ln\frac{4\omega_{D}}{v_{F}|\bm{q}|}-\frac{|\Omega_{m}|}{v_{F}|\bm{q}|})$.
It then follows that in the limit $q\ll k_{F}$,

$D(q)^{-1}\approx\gamma(|\bm{q}|-Q)^{2}+\frac{g^{2}\nu|\Omega_{m}|}{v_{F}Q},$
(13)

resulting in a boson dynamical exponent $z_{b}=2$. A fully self-consistent
solution is obtained by computing the fermion self-energy using $D(\bm{q})$
above. Performing the integrals in the $z_{b}=2$ scaling limit (see
supplemental sections [37]), we obtain
$G^{-1}(\bm{k},i\omega_{n})=G_{0}^{-1}(\bm{k})+{\color[rgb]{0,0,0}\Sigma(\omega_{n})}$,
where

$\Sigma(\omega_{n})={\color[rgb]{0,0,0}i}{\rm
sgn}(\omega_{n})\omega_{0}^{1/2}|\omega_{n}|^{1/2},\
\omega_{0}=\frac{g^{2}Q}{\pi^{2}v_{F}\gamma}.$ (14)

The expressions for $G,D$ are now fully self-consistent: upon feeding back the
fermions to the boson, $\Pi$ is unchanged. Thus, superconducting fluctuations
are Landau overdamped and the fermions are dressed into a non-Fermi liquid.
If, following Hertz[47], we were to integrate out the fermions, the bosonic
sector would be at its upper-critical dimension defined by $d+z=4$, when
$d=2$. Thus, up to logarithmic corrections to scaling, the ordering transition
has mean-field exponents, with $\nu=1/2$. The line of finite temperature
transitions emanates from the quantum critical point as
$T_{c}(g^{2})\sim\left(g^{2}-g_{c}^{2}\right)^{\nu z}$, with unit exponent.
Note that in our toy model Eq.(8), the PDW ordering vector forms a ring, which
renders the whole Fermi surface to be a ‘hot region’111Despite this
peculiarity, fluctuation-induced first order transitions are absent in the
large $N$ limit. However, in the lattice model where there are only limited
number of ordering vectors, there are only finite ‘hot spot’ regions on the
Fermi surface which has NFL behavior.

Discussion.  We have shown that PDW order arises unambiguously when electrons
have sufficiently large repulsive and non-monotonic BCS interactions.
Interactions in the particle-hole channel can certainly destabilize the theory
presented here. However, since ordering tendencies in the particle-hole
channel require finite interaction strength, we expect our theory to remain
robust, at least to the addition of weak interactions in the particle-hole
channel. Other possibilities include Kohn-Luttinger superconductivity, which
also arise from repulsive interactions. However, such states are not present
in the large $N$ limit considered here, and are moreover occur at
exponentially low temperature scales; by contrast the PDW transitions occur at
scales that exhibit power law dependence in the bare interactions of the
system.

We speculate on the relevance of these results to real solids. In microscopic
descriptions of solids, pair-hopping interactions are typically small compared
to density-density interactions[49, 50]. This is not true, however, in low
energy effective descriptions, obtained from integrating out short-distance
modes. In addition, it is somewhat unusual to expect a relatively suppressed
BCS repulsion at short distances. One possible manner to realize such
suppression is to include strong coupling to Holstein phonons. In such a
strong coupling limit, the phonons induce instantaneous short-distance BCS
interactions, which may help screen some of the bare short distance repulsion
coming from say, a Hubbard interaction. This may account for recent studies of
Hubbard-Holstein ladders reporting PDW order[51]. A promising system for
realizing the conditions outlined here for PDW formation are electrons on a
Kagome lattice near the van Hove singularity. In this regime, the electrons
have a peculiar property that short distance Coulomb interactions are
suppressed relative to nearest neighbor repulsion[52]. We shall investigate
the possibility of PDW order in such models, and their relevance to the
phenomena of Kagome metals such as CsV3Sb5, in future studies.

Acknowledgements.  We thank D. Agterberg, A. Chubukov, R. Thomale, Z. Han, P.
Hirschfeld, C. Murthy, S. Kivelson, and J. Sous for helpful discussions. SR
and PN were supported by the Department of Energy, Office of Basic Energy
Sciences, Division of Materials Sciences and Engineering, under Contract No.
DE-AC02-76SF00515. YMW acknowledges the Gordon and Betty Moore Foundation’s
EPiQS Initiative through GBMF8686 for support. AAP is supported by the
Flatiron Institute. The Flatiron Institute is a division of the Simons
Foundation. We thank the participants of the ICTP 2022 workshop on Strongly
correlated matter for enjoyable discussions.

## References

  * Agterberg  _et al._ [2020] D. F. Agterberg, J. S. Davis, S. D. Edkins, E. Fradkin, D. J. Van Harlingen, S. A. Kivelson, P. A. Lee, L. Radzihovsky, J. M. Tranquada, and Y. Wang, The physics of pair-density waves: Cuprate superconductors and beyond, Annual Review of Condensed Matter Physics 11, 231 (2020).
  * Fulde and Ferrell [1964] P. Fulde and R. A. Ferrell, Superconductivity in a strong spin-exchange field, Phys. Rev. 135, A550 (1964).
  * Larkin and Ovchinnikov [1965] A. I. Larkin and Y. N. Ovchinnikov, Nonuniform state of superconductors, Sov. Phys. JETP 20, 762 (1965).
  * Agosta [2018] C. C. Agosta, Inhomogeneous superconductivity in organic and related superconductors, Crystals 8, 285 (2018).
  * Matsuda and Shimahara [2007] Y. Matsuda and H. Shimahara, Fulde–ferrell–larkin–ovchinnikov state in heavy fermion superconductors, Journal of the Physical Society of Japan 76, 051005 (2007).
  * Gurevich [2010] A. Gurevich, Upper critical field and the fulde-ferrel-larkin-ovchinnikov transition in multiband superconductors, Phys. Rev. B 82, 184504 (2010).
  * Cho  _et al._ [2017] C.-w. Cho, J. H. Yang, N. F. Q. Yuan, J. Shen, T. Wolf, and R. Lortz, Thermodynamic evidence for the fulde-ferrell-larkin-ovchinnikov state in the ${\mathrm{kfe}}_{2}{\mathrm{as}}_{2}$ superconductor, Phys. Rev. Lett. 119, 217002 (2017).
  * Fradkin  _et al._ [2015] E. Fradkin, S. A. Kivelson, and J. M. Tranquada, Colloquium: Theory of intertwined orders in high temperature superconductors, Rev. Mod. Phys. 87, 457 (2015).
  * Himeda  _et al._ [2002] A. Himeda, T. Kato, and M. Ogata, Stripe states with spatially oscillating $\mathit{d}$-wave superconductivity in the two-dimensional $\mathit{t}-{\mathit{t}}^{{}^{\prime}}-\mathit{J}$ model, Phys. Rev. Lett. 88, 117001 (2002).
  * Yang  _et al._ [2009] K.-Y. Yang, W. Q. Chen, T. M. Rice, M. Sigrist, and F.-C. Zhang, Nature of stripes in the generalized t-j model applied to the cuprate superconductors, New Journal of Physics 11, 055053 (2009).
  * Raczkowski  _et al._ [2007] M. Raczkowski, M. Capello, D. Poilblanc, R. Frésard, and A. M. Oleś, Unidirectional $d$-wave superconducting domains in the two-dimensional $t-j$ model, Phys. Rev. B 76, 140505 (2007).
  * Berg  _et al._ [2007] E. Berg, E. Fradkin, E.-A. Kim, S. A. Kivelson, V. Oganesyan, J. M. Tranquada, and S. C. Zhang, Dynamical layer decoupling in a stripe-ordered high-${T}_{c}$ superconductor, Phys. Rev. Lett. 99, 127003 (2007).
  * Capello  _et al._ [2008] M. Capello, M. Raczkowski, and D. Poilblanc, Stability of rvb hole stripes in high-temperature superconductors, Phys. Rev. B 77, 224502 (2008).
  * Lee [2014] P. A. Lee, Amperean pairing and the pseudogap phase of cuprate superconductors, Phys. Rev. X 4, 031017 (2014).
  * Wang  _et al._ [2015a] Y. Wang, D. F. Agterberg, and A. Chubukov, Coexistence of charge-density-wave and pair-density-wave orders in underdoped cuprates, Phys. Rev. Lett. 114, 197001 (2015a).
  * Wang  _et al._ [2015b] Y. Wang, D. F. Agterberg, and A. Chubukov, Interplay between pair- and charge-density-wave orders in underdoped cuprates, Phys. Rev. B 91, 115103 (2015b).
  * Wang  _et al._ [2018] Y. Wang, S. D. Edkins, M. H. Hamidian, J. C. S. Davis, E. Fradkin, and S. A. Kivelson, Pair density waves in superconducting vortex halos, Phys. Rev. B 97, 174510 (2018).
  * Partridge  _et al._ [2006] G. B. Partridge, W. Li, R. I. Kamar, Y.-a. Liao, and R. G. Hulet, Pairing and phase separation in a polarized fermi gas, Science 311, 503 (2006).
  * Zwierlein  _et al._ [2006] M. W. Zwierlein, A. Schirotzek, C. H. Schunck, and W. Ketterle, Fermionic superfluidity with imbalanced spin populations, Science 311, 492 (2006), https://www.science.org/doi/pdf/10.1126/science.1122318 .
  * Radzihovsky and Sheehy [2010] L. Radzihovsky and D. E. Sheehy, Imbalanced feshbach-resonant fermi gases, Reports on Progress in Physics 73, 076501 (2010).
  * Cho  _et al._ [2012] G. Y. Cho, J. H. Bardarson, Y.-M. Lu, and J. E. Moore, Superconductivity of doped weyl semimetals: Finite-momentum pairing and electronic analog of the 3he-$a$ phase, Phys. Rev. B 86, 214514 (2012).
  * Bednik  _et al._ [2015] G. Bednik, A. A. Zyuzin, and A. A. Burkov, Superconductivity in weyl metals, Phys. Rev. B 92, 035153 (2015).
  * Wang and Ye [2016] Y. Wang and P. Ye, Topological density-wave states in a particle-hole symmetric weyl metal, Phys. Rev. B 94, 075115 (2016).
  * Li and Haldane [2018] Y. Li and F. D. M. Haldane, Topological nodal cooper pairing in doped weyl metals, Phys. Rev. Lett. 120, 067003 (2018).
  * Wu  _et al._ [2022a] Y.-M. Wu, Z. Wu, and H. Yao, Pair-density-wave and chiral superconductivity in twisted bilayer transition-metal-dichalcogenides 10.48550/ARXIV.2203.05480 (2022a).
  * Wu  _et al._ [2022b] Z. Wu, Y.-M. Wu, and F. Wu, Pair density wave and loop current promoted by van hove singularities in moiré systems 10.48550/ARXIV.2207.11468 (2022b).
  * H.Zhao  _et al._ [2022] H.Zhao, R. Blackwell, S. Ishida, H. Eisaki, A. Pasupathy, and K. Fujita (2022), To appear.
  * Chen  _et al._ [2021] H. Chen, H. Yang, B. Hu, Z. Zhao, J. Yuan, Y. Xing, G. Qian, Z. Huang, G. Li, Y. Ye, _et al._ , Roton pair density wave in a strong-coupling kagome superconductor, Nature 599, 222 (2021).
  * Bhattacharyya and Roy [1995] B. Bhattacharyya and G. K. Roy, The ground state of the penson-kolb-hubbard model, Journal of Physics: Condensed Matter 7, 5537 (1995).
  * Wårdh and Granath [2017] J. Wårdh and M. Granath, Effective model for a supercurrent in a pair-density wave, Phys. Rev. B 96, 224503 (2017).
  * Wårdh  _et al._ [2018] J. Wårdh, B. M. Andersen, and M. Granath, Suppression of superfluid stiffness near a lifshitz-point instability to finite-momentum superconductivity, Phys. Rev. B 98, 224501 (2018).
  * Berg  _et al._ [2010] E. Berg, E. Fradkin, and S. A. Kivelson, Pair-density-wave correlations in the kondo-heisenberg model, Phys. Rev. Lett. 105, 146403 (2010).
  * Loder  _et al._ [2010] F. Loder, A. P. Kampf, and T. Kopp, Superconducting state with a finite-momentum pairing mechanism in zero external magnetic field, Phys. Rev. B 81, 020511 (2010).
  * Loder  _et al._ [2011] F. Loder, S. Graser, A. P. Kampf, and T. Kopp, Mean-field pairing theory for the charge-stripe phase of high-temperature cuprate superconductors, Phys. Rev. Lett. 107, 187001 (2011).
  * Setty  _et al._ [2021] C. Setty, L. Fanfarillo, and P. J. Hirschfeld, Microscopic mechanism for fluctuating pair density wave (2021).
  * Yu  _et al._ [2022] Y. Yu, V. Madhavan, and S. Raghu, Majorana fermion arcs and the local density of states of ${\mathrm{ute}}_{2}$, Phys. Rev. B 105, 174520 (2022).
  * [37] See Online Supplemental Material.
  * Esterlis and Schmalian [2019] I. Esterlis and J. Schmalian, Cooper pairing of incoherent electrons: An electron-phonon version of the sachdev-ye-kitaev model, Phys. Rev. B 100, 115132 (2019).
  * Esterlis  _et al._ [2021] I. Esterlis, H. Guo, A. A. Patel, and S. Sachdev, Large-$n$ theory of critical fermi surfaces, Phys. Rev. B 103, 235129 (2021).
  * Aldape  _et al._ [2022] E. E. Aldape, T. Cookmeyer, A. A. Patel, and E. Altman, Solvable theory of a strange metal at the breakdown of a heavy fermi liquid, Phys. Rev. B 105, 235111 (2022).
  * Patel and Sachdev [2018] A. A. Patel and S. Sachdev, Critical strange metal from fluctuating gauge fields in a solvable random model, Phys. Rev. B 98, 125134 (2018).
  * Wang [2020] Y. Wang, Solvable strong-coupling quantum-dot model with a non-fermi-liquid pairing transition, Phys. Rev. Lett. 124, 017002 (2020).
  * Yang [1989] C. N. Yang, $\eta$ pairing and off-diagonal long-range order in a hubbard model, Phys. Rev. Lett. 63, 2144 (1989).
  * Yang and Zhang [1990] C. N. Yang and S. Zhang, So4 symmetry in a hubbard model, Modern Physics Letters B 04, 759 (1990).
  * Hui and Doniach [1993] A. Hui and S. Doniach, Penson-kolb-hubbard model: A study of the competition between single-particle and pair hopping in one dimension, Phys. Rev. B 48, 2063 (1993).
  * Japaridze and Müller-Hartmann [1997] G. I. Japaridze and E. Müller-Hartmann, Bond-located ordering in the one-dimensional penson - kolb - hubbard model, Journal of Physics: Condensed Matter 9, 10509 (1997).
  * Hertz [1976] J. A. Hertz, Quantum critical phenomena, Phys. Rev. B 14, 1165 (1976).
  * Note [1] Despite this peculiarity, fluctuation-induced first order transitions are absent in the large $N$ limit.
  * Kivelson  _et al._ [1987] S. Kivelson, W.-P. Su, J. R. Schrieffer, and A. J. Heeger, Missing bond-charge repulsion in the extended hubbard model: Effects in polyacetylene, Phys. Rev. Lett. 58, 1899 (1987).
  * Hirsch [1989] J. Hirsch, Bond-charge repulsion and hole superconductivity, Physica C: Superconductivity and its Applications 158, 326 (1989).
  * Huang  _et al._ [2022] K. S. Huang, Z. Han, S. A. Kivelson, and H. Yao, Pair-density-wave in the strong coupling limit of the holstein-hubbard model, npj Quantum Materials 7, 1 (2022).
  * Kiesel and Thomale [2012] M. L. Kiesel and R. Thomale, Sublattice interference in the kagome hubbard model, Physical Review B 86, 121105 (2012).

ONLINE SUPPORTING MATERIAL
Pair density wave order from electron repulsion

Yi-Ming Wu,1 P. A. Nosov,1 Aavishkar A. Patel,2 and S. Raghu1

1 Stanford Institute for Theoretical Physics, Stanford University, Stanford,
California 94305, USA

2 Center for Computational Quantum Physics, Flatiron Institute, New York NY
10010, USA

In this Supplemental Material we (i) comment on the decoupling of the
repulsive BCS interaction, (ii) evaluate the one-loop pair susceptibility at
finite external momenta and frequency (at any temperature), (iii) illustrate
how we evaluate the magnitudes of the PDW gap function below instability
temperature, and (iv) compute the one-loop fermionic self-energy at the QCP.

##  I Hubbard-Stratonovich transformation for repulsive BCS interaction

When the BCS interactions are repulsive, the effective action in Eq. 2 is not
Hermitian. We show here that there are no problems that arise from such an
action. Consider the $0+0$ dimensional version of the theory (which is just an
integral) to illustrate the point. The generalization to the path integral is
trivial. The identity we will use is

$e^{-U\hat{O}^{\dagger}\hat{O}}=\int\frac{d\phi d\phi^{*}}{2\pi
i}e^{-\phi\phi^{*}+i\sqrt{U}\phi^{*}\hat{O}+i\sqrt{U}\hat{O}^{\dagger}\phi},\
\ \hat{O}=c_{\downarrow}c_{\uparrow}$ (S1)

Treating $\phi,\phi^{*}$ as independent fields, their classical equations of
motion are

$\phi=i\sqrt{U}\hat{O},\ \ \phi^{*}=i\sqrt{U}\hat{O}^{\dagger}$ (S2)

at the classical saddle, $\phi^{*}$ is not the complex conjugate of $\phi$.
This naively seems problematic, since the integral over $\phi,\phi^{*}$ then
appears divergent. However, this is false. To see why, it is helpful to work
in a manifestly real representation, defining

$\phi=x+iy,\ \ \phi^{*}=x-iy$ (S3)

Then,

$\displaystyle\int\frac{d\phi d\phi^{*}}{2\pi
i}e^{-\phi\phi^{*}+i\sqrt{U}\phi^{*}\hat{O}+i\sqrt{U}\hat{O}^{\dagger}\phi}=\int\frac{dxdy}{\pi}e^{-(x^{2}+y^{2})+i\sqrt{U}\left(\hat{O}+\hat{O}^{\dagger}\right)x+i\sqrt{U}\left(\hat{O}^{\dagger}-\hat{O}\right)y}$
(S4)

Now, the integrals over x,y can separately be done. This is done by promoting
each of them to a complex number, $x\rightarrow z_{1},y\rightarrow z_{2}$, and
then deforming the contours appropriately to cross their respective saddle
points, neither of which are on the real axis. Thus, the decoupling is
perfectly consistent. This generalizes to the full functional integral.

##  II Cooper pair susceptibility

In this section, we revise the calculation of the one-loop pair susceptibility
at finite external frequency and momenta

$\Pi^{(c)}(q,i\Omega_{m})=T\sum\limits_{n}\int\frac{d^{2}p}{(2\pi)^{2}}\frac{1}{(i\varepsilon_{n}+i\Omega_{m}-\xi_{p+q})(-i\varepsilon_{n}-\xi_{p})}$
(S5)

where $\varepsilon_{n}=2\pi T(n+1/2)$, $\Omega_{m}=2\pi Tm$ are fermionic and
bosonic Matsubara frequencies. We assumed some UV cut-off for Matsubara
frequencies $\omega_{D}$. First, we linearize the dispersion as
$\xi_{p+q}\approx\xi_{p}+v_{F}q\cos\theta$, where $v_{F}$ is the Fermi
velocity, and $\theta$ is the angle between $q$ and $p$ (as usual, this
approximation is sufficient for $q\ll k_{F}$). After integrating over
$\xi_{p}$, we perform the sum over Matsubara frequencies by means of the
series representation of the digamma function. As a result, we arrive at the
following expression for the pair susceptibility:

$\Pi^{(c)}(q,i\Omega_{m})=\nu\ln\left(\frac{2e^{\gamma}\omega_{D}}{\pi
T}\right)-\nu\operatorname{Re}\left\langle\psi\left(\frac{\Omega_{m}+iv_{F}q\cos\theta}{4\pi
T}+\frac{1}{2}\right)-\psi\left(\frac{1}{2}\right)\right\rangle_{\theta}$ (S6)

where $\langle...\rangle_{\theta}=(2\pi)^{-1}\int\limits_{0}^{2\pi}d\theta$
stands for the average over the Fermi surface, and $\gamma$ is the Euler’s
constant. Next, we can make use of the following integral representation of
the digamma function

$\psi\left(z+\frac{1}{2}\right)=\int\limits_{0}^{+\infty}dt\left(\frac{e^{-t}}{t}-\frac{e^{-zt}}{2\sinh
t/2}\right)$ (S7)

After performing the remaining integration over $\theta$, we obtain

$\Pi^{(c)}(q,i\Omega_{m})=\nu\ln\left(\frac{2e^{\gamma}\omega_{D}}{\pi
T}\right)-\frac{\nu}{2}\int\limits_{0}^{+\infty}\frac{dt}{\sinh
t/2}\left(1-J_{0}\left(\frac{v_{F}qt}{4\pi
T}\right)e^{-\frac{|\Omega_{m}|t}{4\pi T}}\right)$ (S8)

After changing variables $t|\Omega_{m}|/(4\pi T)=z$, and defining
dimensionless ratios $\alpha=2\pi T/|\Omega_{m}|$ and
$\beta=v_{F}q/|\Omega_{m}|$, we find

$\Pi^{(c)}(q,i\Omega_{m})=\nu\left[\ln\left(\frac{2e^{\gamma}\omega_{D}}{\pi
T}\right)-\mathcal{K}(\alpha,\beta)\right]$ (S9)

where we introduced the following dimensionless function of $\alpha$ and
$\beta$

$\mathcal{K}(\alpha,\beta)=\alpha\int\limits_{0}^{+\infty}\frac{dz}{\sinh\alpha
z}\left(1-J_{0}(\beta z)e^{-z}\right)$ (S10)

We emphasize that (S9) can be used at finite $T$ as long as $q\ll k_{F}$ (but
for arbitrary ratio $\beta=v_{F}q/|\Omega_{m}|$). Our next goal is to compute
$\mathcal{K}(\alpha,\beta)$ in the limit $\alpha\rightarrow 0$ ($T=0$ limit),
and for fixed $\beta=v_{F}q/|\Omega_{m}|$. This can be done as follows. First,
we observe that

$\mathcal{K}(\alpha,0)=\gamma+2\ln
2+\psi\left(\frac{1+\alpha}{2\alpha}\right)\approx\ln\frac{1}{\alpha}+\ln
2+\gamma+\mathcal{O}(\alpha)$ (S11)

for small $\alpha\ll 1$. Next, we differentiate $\mathcal{K}(\alpha,\beta)$
with respect to $\beta$ and find

$\frac{\partial}{\partial\beta}\mathcal{K}(\alpha,\beta)=\alpha\int\limits_{0}^{+\infty}\frac{dz}{\sinh\alpha
z}\;zJ_{1}(\beta z)e^{-z}$ (S12)

This expression has a well-defined limit $\alpha=0$:

$\lim\limits_{a\rightarrow
0^{+}}\frac{\partial}{\partial\beta}\mathcal{K}(\alpha,\beta)=\int\limits_{0}^{+\infty}dzJ_{1}(\beta
z)e^{-z}=\frac{1}{\beta}\left(1-\frac{1}{\sqrt{1+\beta^{2}}}\right)$ (S13)

Finally, we can integrate back over $\beta$ and use (S11) as the initial
condition

$\int\limits_{0}^{\beta}dy\;\frac{\partial}{\partial
y}\mathcal{K}(\alpha,y)=\mathcal{K}(\alpha,\beta)-\mathcal{K}(\alpha,0)$ (S14)

After expanding both sides in small $\alpha$ and performing the remaining
integral over $y$, we obtain

$\mathcal{K}(\alpha,\beta)=\ln\frac{1}{\alpha}+\gamma+\ln\beta+\frac{1}{2}\ln\left(\frac{\sqrt{\beta^{2}+1}+1}{\sqrt{\beta^{2}+1}-1}\right)+\mathcal{O}(\alpha)$
(S15)

for small $\alpha\ll 1$. Finally, the full $T=0$ expression for
$\Pi^{(c)}(q,i\Omega_{m})$ takes the form

$\Pi^{(c)}(q,i\Omega_{m})=\nu\ln\left(\frac{4\omega_{D}}{v_{F}q}\right)+\frac{\nu}{2}\ln\left(\frac{\sqrt{|\Omega_{m}|^{2}+\left(v_{F}q\right)^{2}}-|\Omega_{m}|}{\sqrt{|\Omega_{m}|^{2}+\left(v_{F}q\right)^{2}}+|\Omega_{m}|}\right)\;,\quad
T=0$ (S16)

with no assumptions on the ratio $|\Omega_{m}|/(v_{F}q)$. If we now consider
the limit $|\Omega_{m}|\ll v_{F}q$, then we find

$\Pi^{(c)}(q,i\Omega_{m})\approx\nu\ln\left(\frac{4\omega_{D}}{v_{F}q}\right)-\frac{\nu|\Omega_{m}|}{v_{F}q}\;,\quad|\Omega_{m}|\ll
v_{F}q$ (S17)

We emphasize that exactly the same result can be easily obtained by taking the
$T=0$ limit directly in the initial expression (S5). Note that
$\Pi^{(c)}(q,i\Omega_{m})$ is related to the boson self-energy
$\Pi(q,i\Omega_{m})$ defined in the main text as
$\Pi(q,i\Omega_{m})=-g^{2}\operatorname{sgn}[V(\mathbf{q})]\Pi^{(c)}(q,i\Omega_{m})$.

##  III Green’s functions below $T_{c}$

The saddle point equations in (6) can be used to extract the information on
the PDW instability temperature $T_{c}$. To obtain the magnitude of the order
parameter when $T<T_{c}$, we need to introduce the anomalous part of the
fermion self energy, and search for convergent solutions. To this end, we add
$g\sum_{k,\bm{q}}(\psi_{\uparrow}(k)\psi_{\downarrow}(-k+q)\Delta(\bm{q})+\text{h.c.})$
into the parent action, such that the total action becomes

$\displaystyle\frac{S}{N}=$ $\displaystyle\int
dr\left[-\eta^{2}g^{2}G^{2}(r)D(-r)+2G(r)\Sigma(-r)+D(r)\Pi(-r)\right]$ (S18)
$\displaystyle+\sum_{k,\sigma}\psi_{\sigma}^{\dagger}(k)[-i\omega_{n}+\xi_{\bm{k}}-\Sigma(k)]\psi_{\sigma}(k)$
$\displaystyle+g\sum_{k,\bm{q}}(\psi_{\uparrow}(k)\psi_{\downarrow}(-k+q)\Delta(\bm{q})+\text{h.c.})$
$\displaystyle+\sum_{q}\bar{\phi}(q)\left[|\lambda(\bm{q})|^{-1}-\Pi(q)\right]\phi(q)$

Since this action is bilinear in both fermion and boson fields, we can
integrate them out. For the fermion fields we work in Nambu space due to the
presence of $\Delta(\bm{q})$, and the resulting action takes the form of
$-\text{tr}\ln\mathcal{F}$, where $\mathcal{F}$ is a matrix given by

$\mathcal{F}_{\bm{k},\bm{k}^{\prime}}=\begin{pmatrix}[-i\omega_{n}+\xi_{\bm{k}}-\Sigma(\bm{k},i\omega_{n})]\delta(\bm{k}-\bm{k}^{\prime})&\Delta(\bm{q})\delta(\bm{k}^{\prime}-\bm{k}+\bm{q})\\\
\bar{\Delta}(\bm{q})\delta(\bm{k}^{\prime}-\bm{k}-\bm{q})&[-i\omega_{n}-\xi_{\bm{k}}+\Sigma(-\bm{k},-i\omega_{n})]\delta(\bm{k}-\bm{k}^{\prime})\\\
\end{pmatrix}$ (S19)

This matrix is diagonal in momentum space only if $\Delta(\bm{q})$ is has a
finite value at $\bm{q}=0$. This is not true for the PDW order, for which
$\Delta(\bm{q})$ is finite at some finite $\bm{q}=\bm{Q}$.

Then the routine procedure of variation leads to the same saddle point
equations as Eq.(6), but with $G(k)$ modified into

$G(k)=\frac{1}{2}\text{tr}\left[\mathcal{F}^{-1}\frac{\delta\mathcal{F}_{\bm{k}^{\prime},\bm{k}^{\prime\prime}}}{\delta\Sigma(k)}\right]$
(S20)

At $T<T_{c}$ we use the solution at $T\geq T_{c}$ as input and solve Eq.(S20)
by sweeping a set of values of $\Delta(\bm{q})$ with $\bm{q}=\bm{Q}$ and
searching for the one which gives convergence after iteration.

##  IV Fermionic self-energy

In this section, we compute the one-loop fermionic self-energy induced by soft
order-parameter fluctuations in a vicinity of a QCP at $g=g_{c}$. In the $T=0$
limit, we obtain

$\displaystyle\Sigma(\omega)$ $\displaystyle=$ $\displaystyle
g^{2}\int\frac{d^{2}qdq_{0}}{(2\pi)^{3}}D(\bm{q},q_{0})G_{0}(\bm{q}-\bm{k}_{F},q_{0}-\omega)$
(S21) $\displaystyle=$ $\displaystyle\frac{g^{2}}{(2\pi)^{3}}\int
dq_{0}qdqd\theta\frac{1}{\gamma(q-Q)^{2}+\frac{g^{2}\nu|q_{0}|}{v_{F}Q}}\frac{1}{i(q_{0}-\omega)-v_{F}q\cos{\theta}}$
$\displaystyle=$ $\displaystyle-i\frac{g^{2}}{(2\pi)^{2}}\int
dq_{0}qdq\frac{1}{\gamma(q-Q)^{2}+\frac{g^{2}\nu|q_{0}|}{v_{F}Q}}\frac{{\rm
sgn}(q_{0}-\omega)}{\sqrt{(q_{0}-\omega)^{2}+(v_{F}q)^{2}}}$

In the $z=2$ scaling limit, the typical frequency $q_{0}^{2}\sim q^{2}\ll
q^{2}$. Therefore, the integral above can be approximated by

$\displaystyle\Sigma(\omega)$ $\displaystyle=$
$\displaystyle\frac{-ig^{2}}{(2\pi)^{2}v_{F}}\int dq_{0}dq\frac{{\rm
sgn}(q_{0}-\omega)}{\gamma(q-Q)^{2}+\tilde{g}|q_{0}|},\ \ \
\tilde{g}\equiv\frac{g^{2}\nu}{v_{F}Q}$ (S22) $\displaystyle=$
$\displaystyle\frac{-ig^{2}}{(2\pi)^{2}v_{F}}\int dq_{0}\frac{{\rm
sgn}(q_{0}-\omega)}{\left(\gamma\tilde{g}|q_{0}|\right)^{1/2}}\left[\frac{\pi}{2}+\arctan{\left(\frac{\gamma^{1/2}Q}{\tilde{g}^{1/2}|q_{0}|^{1/2}}\right)}\right]$
$\displaystyle\simeq$ $\displaystyle\frac{-ig^{2}}{4\pi
v_{F}\left(\gamma\tilde{g}\right)^{1/2}}\int dq_{0}\frac{{\rm
sgn}(q_{0}-\omega)}{|q_{0}|^{1/2}},\ \ \ \ \ \left[\gamma
Q^{2}\gg\tilde{g}|q_{0}|\right]$ $\displaystyle=$
$\displaystyle\frac{ig^{2}}{\pi v_{F}\left(\gamma\tilde{g}\right)^{1/2}}{\rm
sgn}(\omega)|\omega|^{1/2}$

The self energy thus has the form

$\Sigma(\omega)={\rm sgn}(\omega)\omega_{0}^{1/2}|\omega|^{1/2},\ \ \
\omega_{0}=\frac{g^{2}Q}{\pi^{2}v_{F}\gamma\nu}$ (S23)

This NFL behavior survives even beyond one-loop level. In fact, we can obtain
$\Sigma(\omega)$ self consistently, by adding
$\operatorname{sgn}(\omega)|\omega|^{1/2}$ in the fermion Green’s function in
Eq.(S21), and following the same calculations as above, the result is still
given by Eq.(S23).