[1]Ixandra Achitouv

These authors contributed equally to this work.

These authors contributed equally to this work.

[1]Department of Mathematics, Imperial College London and CNRS, Complex
Systems Institute of Paris Île-de-France

2] Oxford Man Institute of Quantitative Finance

3]Department of Mathematics, Imperial College London and the Alan Turning
Institute

# Natural Language Processing for Financial Regulation

<EMAIL_ADDRESS>Dragos Gorduza  Antoine Jacquier  *  [  [

###### Abstract

This article provides an understanding of Natural Language Processing
techniques in the framework of financial regulation, more specifically in
order to perform semantic matching search between rules and policy when no
dataset is available for supervised learning. We outline how to outperform
simple pre-trained sentences-transformer models using freely available
resources and explain the mathematical concepts behind the key building blocks
of Natural Language Processing.

###### keywords:

Language models, Financial Regulation, Natural Language Processing

##  1 Introduction

Over the past ten years, modern natural language processing models have
revolutionised the field of artificial intelligence, transforming how
computers understand and generate language. ChatGPT is one key example of how
AI technology is becoming increasingly important in generating more and more
precise human-like responses to a wide variety of problems, including business
ones. These advancements have significant implications for the financial
industry, where vast amounts of regulatory data and policies must be analysed
and understood in order to comply with laws and regulations.

Indeed, Natural Language Processing (NLP) has gained traction in financial
regulation as a solution for managing and interpreting complex regulatory
information. The ability to effectively process and analyse large amounts of
regulatory text is crucial for compliance officers, risk managers, and other
financial professionals. However, traditional methods of automatic text
processing including keyword searches and dictionaries are both inefficient to
run at the scale required by the amount of files to treat, and costly to set
up since they require system experts to design dictionaries on a case-by-case
basis. Moreover, they are also prone to errors due to lack of coverage which
can expose the user to costly regulatory violations. Modern NLP techniques
have emerged in recent years, mostly based on deep neural networks and after
the seminal breakthrough [1] by Vaswani, Shazeer, Parmar, Uszkoreit, Jones,
Gomez, Kaiser and Polosukhin, who incorporated the attention mechanism to
learn latent semantic links between words in a sentence more accurately. These
improvements, along with orders of magnitude increases in the availability of
datasets and computational power have led to large language model (LLM), a
type of model known for its ability to achieve general-purpose language
understanding and generation such as OpenAI’s GPT (used in ChatGPT), Meta’s
LLaMa or Google’s PaLM. One may however wonder whether LLMs are actually
required for simple NLP tasks such as semantic search, especially due to their
costs, instead of free open resources.

In the present work, we perform a semantic search applied to laws and
regulation using a so called domain adaptation technique with limited amount
of data, bearing in mind limited available resources from small and medium
enterprises. We also recall the historical NLP approaches and explain the
leading concepts in this field. In particular, we focus on semantic search,
which naturally leads to the encoding concepts of a sentence. We thus aim to
fill the gap between well-known NLP concepts and the mathematical frameworks
behind.

This paper is organised as follows: Section 2 is a short historical review on
encoding methods of a sentence into a mathematical object as well as key
machine learning advances. In Section 3 we describe the task of semantic
textual similarity, the different techniques at hand and illustrate them with
pseudo codes. In Section 4, we apply these techniques concretely to financial
regulation problems to improve semantic search matching. Section 5 draws
conclusion from this analysis.

##  2 Historical overview of word embedding approaches

Semantic vectors of words are high-dimensional vectors, and pairwise distances
between two vectors reflect the similarity of words. We briefly review the
main literature on this topic, clarifying the notions of ‘similarity’,
‘meaning’ and ‘correlation between words’.

###  2.1 Deterministic approaches based on word counts

The roots of major modern NLP techniques can be traced back Firth’ quote [2]
_‘You shall know a word by the company it keeps’_. Word co-occurrence
statistic is a simple deterministic approach to obtain a semantic vector
representation of words. Latent Semantic Analysis (HAL) [3] and Hyperspace
Analogue to Language (HAL) [4] are historically the two methods that built
embedded semantic vectors derived from the statistics of word co-occurrence.
LSA derives its vectors from collections of segmented documents, while HAL
makes use of unsegmented text corpora. For segmented documents, a document-
term matrix is used to describe the frequency of terms that occur in a
collection of documents such that rows correspond to the documents and columns
correspond to the terms. Each $(i,j)$ cell, then, counts the number of times
word $j$ occurs in document $i$. Later, Rohde, Gonnerman and Plaut [5]
improved the HAL method introducing the Correlated Occurrence Analogue to
Lexical Semantics (COALS) algorithm, which achieves considerably better
performance through improvements in normalisation and other algorithmic
details.

####  2.1.1 Co-occurrence & Singular Value Decomposition

The deterministic approach proposed in [5] is one of the first methods to
achieve good and consistent results in predicting human similarity judgements
in words embedding. It is a two-step approach:

_Step 1: produce a co-occurrence matrix._ For each word $w_{1}$, count the
number of times another word $w_{2}$ occurs in close proximity to $w_{1}$.
Here ’close proximity’ (or ’windows size’) is a hyperparameter indicating the
number of words to consider before/after the central word $w_{1}$. HAL
suggested to use a 4-word window while COALS proposes 10-word windows. The
counting is done using a weighting scheme whereby if $w_{2}$ is adjacent to
$w_{1}$ it receives the maximal weight, and the weight is proportional to the
length separation of the central word $w_{1}$ (ramped window). This results in
an $N\times N$ co-occurrence matrix where $N$ represents the number of words
in the whole corpus. One could in principle stop here, where each row of the
co-occurrence matrix can thus be associated with a word. With this procedure,
HAL [4] demonstrated semantic similarity between any desired pair of words. A
few tricks were also proposed in the computation of the co-occurrence matrix,
for instance putting a threshold to the number of co-occurrence count to
remove frequent words such as ‘the’ or ‘a’.

_Step 2: Dimension reduction of word vectors._ From the co-occurrence count of
words the resulting embedded vectors should have at least the size of the
words in the corpus111In fact it has a $2N$ dimension in HAL where for every
word in the target vocabulary, there is both a row and a column containing
relevant values. For instance, the row may contain co-occurrence information
for words appearing before the word under consideration, while the column
contains co-occurrence information for words following it. This (row,column)
pair may be concatenated so that, given an $N\times N$ co-occurrence matrix, a
co-occurrence vector of length $2N$ is available. In COALS the distinction
between left/right of the central word is ignored.. In HAL, the authors
proposed to keep a relatively small number of principal components of the co-
occurrence matrix. For instance reducing the vector size by eliminating all
but the $k$ columns with the highest variance. In [5] the reduction of
dimension is based on Singular Value Decomposition (SVD) of the normalised co-
occurrence matrix (this was also proposed in for latent semantic analysis in
[3]).

###  2.2 Machine Learning

Over the last decade, Machine Learning algorithms contribution to NLP have
brought significant progress. Among these, Word2vec in 2013 by Mikolov, Chen,
Corrado and Dean [6], Glove in 2014 by Pennington, Socher and Manning [7],
BERT in 2018 by Devlin, Chang, Lee and Toutanova [8] and SBERT in 2019 by
Reimers and Gurevych [9] are the most popular ones, which we shall use in this
project. The key idea is to perform a non-linear regression where the
coefficients are adjusted by minimising a loss function (or maximising a
likelihood function). In the case of NLP, the coefficients/weights are the
embedded vectors.

####  2.2.1 Skip-gram and CBOW

To understand the Skip-gram model, we briefly review some fundamental concepts
of probabilistic language modelling where the goal is to compute the
probability $\mathbb{P}(\mathrm{w})$ of a sequence of words
$\mathrm{w}:=(w_{0},\ldots,w_{n})$ occurring; we also write
$\mathrm{w}_{i:j}:=(w_{i},\ldots,w_{j})$ and omit $i$ whenever $i=0$. A
similar goal is to compute the conditional probability of an upcoming word
$\mathbb{P}(w_{j}|\mathrm{w}_{j-1})$. The computation of these probabilities
rely on conditional probabilities where the general chain of rule can be
expressed as

$\mathbb{P}(\mathrm{w})=\mathbb{P}(w_{0})\mathbb{P}(w_{1}|w_{0})\mathbb{P}(w_{2}|\mathrm{w}_{:1})\cdots\mathbb{P}(w_{n}|\mathrm{w}_{:n-1}).$

To estimate this probability we count from a corpus

$\mathbb{P}(w_{j}|\mathrm{w}_{:j-1})=\frac{\\#\mathrm{w}_{:j}}{\\#\mathrm{w}_{:j-1}}.$

For sentences with more than a few words, the probability that a complex
sentence repeats itself often enough to be able to compute statistical
properties is unlikely. To overcome this, one usually assumes a pseudo-
Markovian approximation (the $k$-gram model) 222strictly speaking it is not
Markovian for $k\neq 1$ because we keep a memory of the $k$ previous words,
whereby
$\mathbb{P}(w_{j}|\mathrm{w}_{:j-1})\approx\mathbb{P}(w_{j}|\mathrm{w}_{j-k:j-1})$
for some $k\leq j$. For instance for $k=1$ we get the properly Markov bi-gram
model $\mathbb{P}(w_{j}|\mathrm{w}_{:j-1})\approx\mathbb{P}(w_{j}|w_{j-1})$.
Given a sentence, computing for each word the probability that it occurs given
the previous words is similar to filling a normalised co-occurrence matrix.

Skip-gram models are generalisations of $n$-gram models, in which the
components do not need to be consecutive, but may leave gaps that are skipped
over [10, 11]. For instance, the adjacent words for the sentence: _‘I hit the
blue ball’_ are $\\{\emph{I hit},\emph{hit the},\emph{the blue},\emph{blue
ball}\\}$ for a bi-gram and $\\{\emph{I hit},\emph{I the},\emph{I
blue},\emph{hit the},\emph{hit blue},\emph{hit ball},\emph{the blue},\emph{the
ball}\\}$ for a 2-skip-bi-gram. Finally, in Word2vec [12], the definition of
the Skip-gram architecture is to predict surrounding words given a specific
word, which formally corresponds to computing
$\mathbb{P}(\mathrm{w}_{n-j:n+j}|w_{n})$. while the Continuous Bag of Words
(CBOW) architecture predicts the current word based on the context:
$\mathbb{P}(w_{j}|\mathrm{w}_{:j-1})$. These two models are illustrated in
[12, Figure 1].

####  2.2.2 Link with the embedding vectors

In the Skip-gram originally presented in [12], the training objective is to
learn word vector representations that are good at predicting the _close-by_
words $w(t-2),w(t-1),w(t+1),w(t+2)$ from an input word $w(t)$. This is
achieved by maximising the average log likelihood probability, equivalently by
minimising the loss function

$\mathcal{J}(\Theta):=-\frac{1}{T}\sum^{T}_{t=1}\sum_{-c\leq j\leq c,j\neq
0}\log\mathbb{P}(w_{t+j}|w_{t}),$ (1)

where $c$ is the size of the training context, $(w_{1},\cdots,w_{T})$ is a
sequence of training words and $\Theta$ is the vector representation of words
that the algorithm tries to optimise. Note that every word $w$ appears both as
a central word (input $w_{I}$) and as a context word (output $w_{O}$) so that
$\Theta\in\mathbb{R}^{2\mathfrak{w}}$ where $\mathfrak{w}$ is the size of the
vocabulary333the final embedded word $w$ is simply the average of its input
$w_{I}$ and output $w_{O}$ vector representations.. In the basic Skip-gram
formulation, a softmax function is proposed to model the loss function, namely

$\mathbb{P}(w_{O}|w_{I})=\frac{\exp\left(v_{w_{O}}^{\top}v_{w_{I}}\right)}{\sum_{w=1}^{\mathfrak{w}}\exp\left(v_{w}^{\top}v_{w_{I}}\right)},$
(2)

where $w_{O}$ and $w_{I}$ are the output and input words, and, given a word
$w$, $v_{w_{I}}$ and $v_{w_{O}}$ are its input and output vector
representations. The loss function is then minimised via gradient descent: the
gradient of the likelihood (with respect to $v_{w_{I}}$) can be computed
easily and is equal to zero if and only if

$v_{w_{O}}=\left[\sum_{w=1}^{\mathfrak{w}}\exp(v_{w}^{\top}v_{w_{I}})\right]^{-1}\sum_{w=1}^{\mathfrak{w}}v_{w}\exp(v_{w}^{\top}v_{w_{I}}),$
(3)

which is the expectation of $v_{w_{O}}$ occurring given the context word
$v_{w_{I}}$.

In practice, to improve computation efficiency, a hierarchical softmax is
proposed in [12, Equation (3)]), which uses a binary tree representation such
that the computing time cost reduces from $\mathfrak{w}$ to
$\log(\mathfrak{w})$. An alternative technique to improve efficiency is to
sub-sample $k$ random words and compute their probabilities to appear in the
context of the central words. This is known as negative sampling where
$\rm\log\mathbb{P}(w_{t+j}|w_{t})$ in (1) is replaced by

$\log\sigma\left(v_{w_{O}}^{\top}v_{w_{I}}\right)+\sum_{j=1}^{k}\mathbb{E}_{w_{j}\sim
P_{n}(w)}\left[\log\sigma\left(-v_{w_{j}}^{\top}v_{w_{I}}\right)\right],$ (4)

where $\sigma(\cdot)$ is the sigmoid function and the sum runs over a random
sample of $k$ words (rather than $\mathfrak{w}$) based on their frequencies
around the center word. Thus, the task is to distinguish the target word
$w_{O}$ from draws from noise distribution $P_{n}(w)$ using logistic
regression. The noise distribution $P_{n}(w)$ in (4) is a free parameter and
[12] found that $P_{n}(w)=u(w)^{3/4}/Z$, where $u(w)$ is a unigram and $Z$ is
just the normalisation constant, outperformed computational efficiency
significantly over the uniform distributions. This avoids summing over all
words in the denominator of (2). In [12], typical values for $k$ lie in the
range $[5,20]$ for small training datasets, while for large datasets $k$ can
be as small as $[2,5]$. An implementation of this model can be found at
https://github.com/chrisjmccormick/word2vec_commented/blob/master/word2vec.c.
Improvements of the CBOW and Skip-gram models have been proposed but still
rely on the same fundamental idea of co-occurrence statistics of words from
[7].

####  2.2.3 Transformer and attention mechanism

Another major step was archived using multi-layer bidirectional Transformer
encoders [1] for the training architecture, and was used in BERT [8]
(_Bidirectional Encoder Representations from Transformers_). A main difference
with Word2vec is that BERT is a non-static model with not just one hidden
layer. Another key difference is that there is no window-size limitation of
n-words in BERT to consider the context of the word to encode. The technique
employed instead is known as the _attention mechanism_ and is the building
block of a transformer architecture. Transformers were originally designed for
translation and they are now the state of the art architecture in NLP. The
transformer model architecture can be seen in [1, Figure 1].

Transformers can be summarised as encoder-decoder tools, where a sentence is
fed to the encoder and the decoder generates another sentence; the whole
system is built by stacking several layers or encoder/decoder. Each encoder
has two sub-layers, a multi-head self-attention layer and a position-wise
fully connected feed-forward network, and receives a list of vectors as input,
passes it into a self-attention layer, then into a feed-forward neural
network, and finally sends out the output upwards to the next encoder. The
decoder itselg has tree layers: in addition to the two sub-layers of the
encoder layer, the decoder inserts a third sub-layer, which performs multi-
head attention over the output of the encoder stack and helps the decoder
focus on relevant parts of the input sentence.

The Self-attention mechanism is the building block of a Transformer and
captures the relationship between the different words in a sentence. All
attention mechanisms in a Transformer are split into independent heads whose
outputs are concatenated before being further processed. A self-attention
mechanism computes a score value between a query with keys, and reweighs the
value. All heads’ values will be used through a scale dot architecture to
update the attention weight matrix. A self-attention head (assuming one single
head) can summarised as follows [13]:

  * •

For each word $i\in\\{1,\ldots,L\\}$ in a sentence of length $L$, return its
positional embedding vector $\mathrm{w}_{i}$ of length $d_{e}$, which is one
of the hyperparameters. In [1] the base case is $d_{e}=512$ while for example
in GPT2 it is $768$. The size of the embedding impacts the probability that
two random embedded words are correlated or not (if their scalar product is
close to $\pm 1$ or to $0$ respectively).

  * •

Stack each embedding into a matrix $\mathbf{X}\in\mathbb{R}^{L\times d_{e}}$.
For instance if we consider each English word as a token, the sentence ’I hit
the blue ball’ has $L=5$ tokens, each of them encoded in a $512$-dimensional
embedding vector.

  * •

Compute the query, key and value matrices Q, K and V by projecting
$\mathbf{X}$ in the (subspace) representations of the packed queries, keys and
values:

$\mathbf{Q}:=\mathbf{X}\mathbf{W}_{q},\qquad\mathbf{K}:=\mathbf{X}\mathbf{W}_{k},\qquad\mathbf{V}:=\mathbf{X}\mathbf{W}_{v},$

where $\mathbf{W}_{q}$, $\mathbf{W}_{k}$, $\mathbf{W}_{v}$ are weights
matrices of length $d_{e}\times d_{w}$ trained during the training process
(randomly initialised) and their dimensions depend on the architecture. For
simplicity, in one single-head attention $d_{w}=d_{e}$, while for multiple-
head ($8$ in [1]), $d_{w}=d_{e}/N_{\rm head}=512/8=64$. Multiple-head
attention algorithm split multiple query, key and value ‘heads’ in order to
improve the performance of the algorithm. This split reshapes weight matrices
into $\mathbf{W}_{q}^{i}$, $\mathbf{W}_{k}^{i}$, $\mathbf{W}_{v}^{i}$, of
dimensions $d_{e}\times d_{w}$. In the base case of [1], $d_{w}=64$ was
selected after hyperparameter tuning as was the length of the embedded vectors
$d_{e}$. Larger values of the hyperparameters such as $d_{w}$ or $d_{e}$
provide better performance [1, Table 4], but require more computational
resources.

  * •

Compute a self-attention score of each word of the input sentence against the
others, by taking the dot product between query and key vectors of the
respective words. These scores are then normalised and passed into a softmax
function. The resulting attention-weighting matrix of size $L\times L$
represents the correlation between word $i$ and word $j$. Finally, the
attention function or self-attention head takes the attention-weighting matrix
and multiply it by the value matrix $\mathbf{V}$, defined as

$\rm
Attention(\mathbf{Q},\mathbf{K},\mathbf{V})=\mathrm{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^{\top}}{\sqrt{d_{w}}}\right)\mathbf{V},$
(5)

resulting in an updated vector representation of the contextualised token
$\mathbf{V}$.

The whole transformer itself uses $h$ attention heads ($h=8$ in [1]) and is
summarised in Algorithm 1 to allow the model to jointly represent information
from different representation subspaces at different positions. The reweighing
“values” from the heads are passed through another densely connected layers.
The outputs of these attention heads are concatenated into a matrix
$\mathbf{O}$ of dimensions $L\times d_{e}$. This matrix is multiplied by an
output weight matrix $\mathbf{W}_{o}$ of dimension $d_{e}\times d_{e}$. The
result is $\widetilde{\mathbf{X}}$, an updated representation of $\mathbf{X}$
the input matrix of the embedded tokens with
$\widetilde{\mathbf{X}}\in\mathbb{R}^{L\times d_{e}}$.

Algorithm 1 Multiheaded Attention Layer

$\mathbf{X}$, $\mathbf{W}_{q}$, $\mathbf{W}_{k}$,
$\mathbf{W}_{v},\mathbf{W}_{o}$

for $i=1$ to $h$ do (Split $\mathbf{W}_{q}$, $\mathbf{W}_{k}$, and
$\mathbf{W}_{v}$ into $h$ heads
$(\mathbf{Q}_{i},\mathbf{K}_{i},\mathbf{V}_{i})_{i=1,\ldots,h}$)

$\mathbf{Q}_{i}\leftarrow\mathbf{X}\mathbf{W}_{q}^{i}$;
$\mathbf{K}_{i}\leftarrow\mathbf{X}\mathbf{W}_{k}^{i}$;
$\mathbf{V}_{i}\leftarrow\mathbf{X}\mathbf{W}_{v}^{i}$

$\mathrm{output}_{i}\leftarrow\mathrm{Attention}(\mathbf{Q}_{i},\mathbf{K}_{i},\mathbf{V}_{i})$;

$\mathbf{O}\leftarrow\text{Concatenate}(\mathrm{output}_{1},\ldots,\mathrm{output}_{h})$

$\widetilde{\mathbf{X}}\leftarrow\mathbf{O}\mathbf{W}_{o}$

$\widetilde{\mathbf{X}}$

The transformer model has been very successful and is at the core of many
machine learning applications: for NLP, the BERT family [1], in biology, the
Generative Pre-trained Transformer (GPT) family to predict molecular
structures [14], in computer vision, to extract image information [15]. Recent
models differ from their predecessors primarily by their size (BERT has 310
million parameters, GPT3 has 175 billion [16], PALM has 540 billion). The
other chief difference lies in which part of the encoder-decoder framework of
the transformer architecture is leveraged. BERT uses the entire encoder-
decoder while models like the GPT family use the only decoder. The classical
BERT architecture is trained on two tasks [8]. The first is sentence
entailment, where the input is one sentence and the model is asked to predict
whether the question proposed as the output is logically entailed by the first
or not. The second task is Masked Language Modelling, where a random $15\%$
subset of tokens in the entire training corpus is replaced by a [MASK] token
and the BERT model is asked to predict the most likely candidate replacement.
Both these pre-training approaches yield a very powerful model that can be
fine-tuned on a downstream task.

##  3 Semantic textual similarity search

To perform semantic similarity search, a classical approach would be to feed
an input sentence or text to the BERT transformer network which produces
contextualised word embeddings for all input tokens in the text. Then, via a
pooling layer (such as mean-pooling), the average of the contextualised word
embeddings would return a fixed-sized output representation vector. In the
BERT-base model the dimension is 768; similarly to the previous discussion,
larger values provides better accuracy but at the cost of computational memory
and time. This sentence/text could be directly compared with another pair
using for instance cosine similarity between the two vectors. However this
would often yield poor sentence embedding compared to GloVe [9] and to
expensive computational time. For instance in [9], computing the most similar
sentence pairs from $1,000$ sentences took $65$ hours with BERT. We have also
tried to perform some fine-tuning and further train BERT on our corpus and
indeed found the computational time to be significantly larger and the
semantic search score high even for non matching (policy, rule) pair.

To palliate this, SBERT (Sentence-BERT) uses a BERT architecture with siamese
and triplet networks [17] that is able to derive semantically meaningful
sentence embeddings. This adds a pooling operation to the output of BERT and
fine-tune the latter by adding a training dataset with pairs of sentences. The
cosine similarity between a query vector $\mathbf{Q}$ and a policy vector
$\mathbf{P}$ can be defined as

$\mathcal{S}_{c}(\mathbf{Q},\mathbf{P}):=\frac{\mathbf{Q}^{\top}\mathbf{P}}{\|\mathbf{Q}\|\|\mathbf{P}\|},$
(6)

which converges to one as the similarity increases. This allows SBERT networks
to be fine-tuned and to recognise the similarity of sentences. Different
versions of loss functions can be used; for example, we later use the all-
MiniLM-L6-v2, trained on a dataset of over one billion training pairs. These
models can be freely accessed using Sentences Transformers (www.sbert.net) to
perform text embeddings directly from a large collection of pre-trained models
tuned for various tasks.

Our goal here is to link rules issued by an organisation (such as governments
or financial regulators) to financial institutions policies (banks in
particular). Financial rules are often updated and financial institutions are
required to issue new policies addressing these changes. It is extremely
challenging for large institutions to keep track of regular updates and to
ensure their policies match the latest rules. Using NLP, we perform here
semantic search between query sentences (rules) and answers (policies) written
in a corpus as follows:

  * •

Clean corpus and rules from meaningless symbols (bracket, hashtag, special
character, …).

  * •

For each query, split the corpus into sentences or paragraphs, depending on
the data and the query size. Sentence-by-sentence comparison similarity will
be more precise than pooling many sentences into paragraphs.

  * •

Use a pre-trained model such as BERT to tokenise each word of the sentence,
and map sentences to embedded vectors, usually via mean-pooling of each
embedded token. The size of any embedded token depends on the pre-trained
model. One can also use pre-trained model such as all-MiniLM-L6-v2 and embed
the sentence directly using sentence-transformer library.

  * •

Use cosine similarity (6) as a metric between an embedded rule query and the
embedded policies to find the best match. By requiring that a match is
positive above a threshold (say $\mathcal{S}_{c}>0.7$), we can easily extract
from a policy corpus which sentences match the rule (if any).

This powerful pipeline can however be improved. Using pre-trained models to
encode our sentences to vectors may sound sensible, but they are trained on
generic corpus and are not familiar with domain-specific targets–such as
financial regulation–with high accuracy. One may instead completely re-train
BERT on a domain-specific corpus, as done in [18]; this however requires huge
amount of training data ($3.3$ billion tokens), which is extremely costly.
Instead–and this is the state of the art– training can be performed using an
Adaptative Pre-Training, described at www.sbert.net; this was successfully
applied in various domains, such as for the bio-medical language [19]. It can
then be archived with Sentence-Transformers frameworks and the Hugging Face
Transformers library, specifically built for NLP applications huggingface.co.

###  3.1 Classical Domain Adaptation (DA) Pre-Training method

The DA Pre-Training method can be split into two steps:

  * •

Step 1: further train a pre-trained model for words embeddings such as BERT,
SBERT or all-MiniLM-L6-v2 using a context specific corpus.

  * •

Step 2: fine-tune the resulting model based on an existing training dataset
(such as paired sentences), similarly to SBERT but on the domain-specific
dataset.

Before performing Step 1, one can also add up domain-specific tokens to the
existing model, although given the limited amount of datasets at hand, this
may not always be possible. To perform Step 1, known as Pre-Training on Target
Domain, we use the Mask Language Modelling (MLM) approach [20], which masks a
random fraction of tokens in a sentence and the transformer then tries to
guess what is missing. Alternative approaches include Transformer-based
Denoising AutoEncoder (TSDAE) [20]. While MLM outputs a token vector (mask
word), the TSDAE encoder is fed with noisy sentences which the decoder uses to
predict the full original sentences. To perform Step 2, we need to specify the
loss function and the training set. Many generic training sets with labelled
paired sentences are available (huggingface.co/datasets), but we do not use
them since wish to target our fine-tuning on (rules, policies) pairs. For the
loss function, we use the Multiple Negatives Ranking (MNR) Loss detailed in
[21, Section 4] and reviewed below.

The idea of Multiple Negatives Ranking (MNR), introduced in [21], is to
suggest responses to a question embedded in a vector $\mathbf{X}$ maintaining
some kind of memory throughout the course of a dialogue, which translates to
finding the probability $\mathbb{P}(\mathbf{Y}|\mathbf{X})$, where
$\mathbf{Y}$ is the embedded vector answer. This probability is used to rank
possible answers $\mathbf{Y}$ given an input question $\mathbf{X}$. Bayes’
theorem and the total law of probability indicate that this probability can be
expressed as

$\mathbb{P}(\mathbf{Y}|\mathbf{X})=\frac{\mathbb{P}(\mathbf{X},\mathbf{Y})}{\sum_{k}\mathbb{P}(\mathbf{X},\mathbf{Y}_{k})},$
(7)

and the joint probability $\mathbb{P}(\mathbf{X},\mathbf{Y})$ is estimated
using a neural network scoring function $\mathcal{S}$ such that

$\widehat{\mathbb{P}}(\mathbf{X},\mathbf{Y}):=\exp\left\\{\mathcal{S}(\mathbf{X},\mathbf{Y})\right\\}.$

In practice the denominator in (7) (equal to $\mathbb{P}(\mathbf{X})$) is
approximated by sampling $K$ responses from the training corpus with
$\widehat{\mathbb{P}}(\mathbf{X}):=\sum_{k=1}^{K}\mathbb{P}(\mathbf{X},\mathbf{Y}_{k})$,
leading to the approximate probability used to train the neural network:

$\widehat{\mathbb{P}}(\mathbf{Y}|\mathbf{X}):=\frac{\exp\left\\{\mathcal{S}(\mathbf{X},\mathbf{Y})\right\\}}{\sum_{k=1}^{K}\exp\left\\{\mathcal{S}(\mathbf{X},\mathbf{Y}_{k})\right\\}}.$

Having a training dataset with (question, answer) pairs, each embedded
question $\mathbf{X}_{i}$ is paired with the embedded answer $\mathbf{Y}_{i}$
and all other $\mathbf{Y}_{j}$ for $i\neq j$ is treated as a negative
candidate for question $\mathbf{X}_{i}$ (so $K-1$ negatives). The goal of the
training is to minimise the loss function

$\mathcal{J}(\mathbf{X},\mathbf{Y},\Theta):=-\frac{1}{K}\sum_{i=1}^{K}\log\widehat{\mathbb{P}}(\mathbf{Y}_{i}|\mathbf{X}_{i}),$

where $\Theta$ gathers the word embeddings and the neural network parameters.
To compute the score, the authors in [21] represent the input question
tokenised into a word sequence $\mathbf{X}$ and responses $\mathbf{Y}$ as
fixed-dimensional input features, extracting $n$-gram features from each.
During training, they learn a $d$-dimensional embedding for each $n$-gram
jointly with the other neural network parameters. To represent sequences of
words, they combine $n$-gram embeddings by summing their values. This bag of
$n$-grams representation is denoted as $\Psi(\mathbf{X})\in\mathbb{R}^{d}$.
Then a feedforward scoring model takes the $n$-gram representation of a
question and a response, and computes a score. For instance in the dot-product
architecture from [21, Figure 3], $\Psi(\mathbf{X})$ and $\Psi(\mathbf{Y})$ go
to two separate $\tanh$ activation layers returning the encoded $\mathbf{X}$
and $\mathbf{Y}$ as $\mathbf{h}_{x}$, $\mathbf{h}_{y}$ and performing
$\mathcal{S}(\mathbf{X},\mathbf{Y})=\mathbf{h}_{x}^{\top}\mathbf{h}_{y}$.
Using the MNR Loss, the score is computed using at first the chosen pre-
trained model and the default scoring is computed via cosine similarity in
(6).444see github.com/UKPLab/sentence-
transformers/blob/master/sentence_transformers/losses/MultipleNegativesRankingLoss.py
for details The key issue and challenge for this work lies in the absence of a
training dataset for policies and rules. While unsupervised text embedding
learning performs rather poorly in learning on domain-specific concepts
without fine tuning, we tested an approach to perform an Adaptative Pre-
Training without using the training dataset we generated in Section 4.1. This
method generates pseudo-labeling pairs for fine tuning and we describe it in
the next section.

###  3.2 Domain Adaptation with Generative Pseudo-Labeling

Alternatively to the approach described above, we also tested the unsupervised
fine-tuning with Generative Pseudo-Labeling (GPL) [22]. This method can be
used to perform Step 2 from Section 3.1. In our context we wish to investigate
how it improves Step 2 over a fine-tuning fitting method that uses labelled
(rules, policies) pairs. The idea behind GPL can be condensed into four steps,
that we now describe. Below, we shall use calligraphic letters to denote sets
(of paragraphs for example) as opposed to standard letters for elements, for
example $\mathcal{P}^{-}:=\\{P^{-}_{1},P^{-}_{2},\ldots\\}$.

####  3.2.1 GPL Step 1: Generating queries from a domain specific corpus

For each paragraph of the corpus, we generate queries using a
$\mathrm{T5}$-encoder-decoder model [23] similarly to the architecture of [1]
discussed in Section 2.2.2. This query generator, which can easily be called
again from Sentences-Transformer models, yields positive (query, answer) pairs
and is summarised in Algorithm 2.

Algorithm 2 Generator

input paragraph $P^{+}$, vocabulary size $V$, length of the output query
paragraph $L_{q}$

$G\leftarrow\mathrm{T5}(P^{+})$ ($G\in\mathbb{R}^{V\times L_{q}}$: matrix of
probability distributions)

$Q\leftarrow(\operatorname*{arg\,max}(G_{i}))_{i=1,\ldots L_{q}}$

$Q\leftarrow\mathrm{Decode}(Q)$

(Q,$P^{+}$)

####  3.2.2 GPL Step 2: Negative mining using dense retrieval

This step finds sentences in the text that share many similar words. For each
query or paragraph, there is now a positive pair. In addition, for each of the
generated queries, GPL retrieves $M$ negative passages that are similar but
not a match to the query, which is known as a negative mining process. At the
end of this step, we obtain triplets of positive and negative passages
associated with a query. This step makes use of a function DenseRetrieval
which is detailed in Algorithm 3.

Algorithm 3 Dense Retrieval

Remaining set
$\mathcal{P}=\\{P_{1},\ldots,P_{|\mathcal{P}|}\\}\setminus\\{P^{+}\\}$ of
paragraphs; user query $Q$; $M\leq|\mathcal{P}|$: number of negative passages;

$\mathrm{\mathcal{P}^{-}},\mathcal{D}\leftarrow\\{\\},\\{\\}$

$E\leftarrow(\mathrm{T5}(P_{i}))_{i=1,\ldots,|\mathcal{P}|}$ (Calculate an
embedding $E$ for each paragraph)

$E_{Q}\leftarrow\mathrm{T5}(Q)$ (Calculate an embedding $E_{Q}$ for from the
query $Q$)

$\mathcal{D}\leftarrow({\mathcal{S}_{c}(E_{i},E_{Q}),E_{i}})_{i=1,\ldots|E|}$
(with $\mathcal{S}_{c}(\cdot)$ defined in (6))

$\mathcal{D}\leftarrow\mathrm{sort}(\mathcal{D})$ (Sort by cosine similarity)

$\mathcal{P}^{-}\leftarrow(D_{1},\ldots,D_{M})$

$(Q,P^{+},\mathcal{P}^{-})$

####  3.2.3 GPL Step 3: Pseudo labelling

Use a crossencoder [24] to score all query-positive passage and query-negative
passages. Note that some negative passage might actually be positive pairs.
This step scores the triplets composed of our initial paragraph $P^{+}$, our
generated query based on the initial paragraph $Q$, one output $P^{-}_{i}$
from our set of negative samples $\mathcal{P}^{-}$.

Algorithm 4 Pseudo Labelling

$(Q,P^{+},\mathcal{P}^{-})$, crossencoder, $M$

for $i=1$ to $M$ do

$\mathrm{margin}_{i}\leftarrow\mathrm{crossencoder}(Q,P^{+})-\mathrm{crossencoder}(Q,P^{-}_{i})$

$(Q,P^{+},P^{-}_{i},\mathrm{margin}_{i})_{i=1,\ldots,M}$

####  3.2.4 GPL Step 4: Training/Tuning the Transformer model to identify the
differences between positive and negative passages, using a crossencoder model

The latter compare the embeddings of the passages by generating similarity
scores for both positive and negative pairs. Given a transformer model that we
wish to train using GPL and the output tuples from the previous step, we will
train the said transformer by backpropagation between the margin computed in
Algorithm 4 and the predicted margin we extract. This process is optimised
using an MSE loss function applied to the margin

$S_{\mathrm{MSE}}:=\left|X_{Q}^{\top}\left(X_{P^{+}}-X_{\mathcal{P}^{-}_{i}}\right)\right|,$
(8)

with $X_{Q},X_{P^{+}},X_{\mathcal{P}^{-}_{i}}$ the embedded vectors of the
query, positive passage, and negative passage i.

Algorithm 5 Training loop of Transformer using GPL

$(Q,P^{+},P^{-}_{i},\mathrm{margin}_{i})_{i=1,\ldots,M}$

for $i=1$ to $M$ do

$(X_{Q},X_{P^{+}},X_{P^{-}_{i}})\leftarrow\mathrm{Transformer}(Q,P^{+},P^{-}_{i})$

$\mathrm{predicted}_{\mathrm{sim}^{+}}\leftarrow\mathcal{S}_{c}(X_{Q},X_{P^{+}})$
(with $\mathcal{S}_{c}(\cdot)$ defined in (6))

$\mathrm{predicted}_{\mathrm{sim}^{-}_{i}}\leftarrow\mathcal{S}_{c}(X_{Q},X_{P^{-}_{i}})$

$\mathrm{predicted}_{\mathrm{margin}_{i}}\leftarrow\mathrm{predicted}_{\mathrm{sim}^{+}}-\mathrm{predicted}_{\mathrm{sim}^{-}_{i}}$

$\mathrm{Loss}_{i}\leftarrow
S_{\mathrm{MSE}}(\mathrm{margin}_{i},\mathrm{predicted}_{\mathrm{margin}_{i}})$
defined in (8))

$\mathrm{Output}_{i}\leftarrow\mathrm{backpropagate}(\mathrm{Loss}_{i})$

$\mathrm{Output}$

###  3.3 To summarise

Both supervised and unsupervised domain adaptation methods use a pre-trained
model and further train it by comparing (query, answer) pairs or triplets
(negative answer). These pairs and triplets are in the classical supervised
method given by the dataset used to perform the domain adaptation (corpus of a
specific domain), but it may also be generated. That is the GPL approach for a
fully unsupervised domain adaptation. Below, we explore the results of
performing an Adaptative Pre-Training method and a purely unsupervised
approach, using GPL. To compare our results and to perform supervised fine-
tuning we need a pseudo training/validation dataset. We describe in the next
section how we create it.

##  4 Application to semantic matching for financial regulation

###  4.1 Creation of a pseudo training and validation dataset

To create our validation dataset we use a mean assemble average of N pre-
trained models (in what follows N=10)555 multi-qa-mpnet-base-cos-v1,
sentence-t5-xl, multi-qa-distilbert-cos-v1, msmarco-bert-base-dot-v5, all-
distilroberta-v1, all-MiniLM-L12-v2, distiluse-base-multilingual-cased-v2,
all-mpnet-base-v2, stsb-distilbert-base, bert-base-nli-mean-tokens. These
models can be found at www.sbert.net/docs/pretrained_models.html for the task
of semantic search between rules and policies. We run these N sentence-
transforming models on a catalogue of rules and financial policies provided by
FinregE (https://finreg-e.com), a company providing clients in financial
services with a software focusing on current and future regulations, using ML
and AI tools to identify and interpret regulatory requirements and to
integrate compliance workflows for action and compliance management.

The dataset of rules is publicly available from Financial Conduct Authority
(FCA) Rulebook 2022. After cleaning and splitting the rules into sentences of
length 200, we obtain about 50,000 sentences, composed of 21,914 rule IDs. The
policies’ dataset was obtained from FinregE and is composed of 2,374 policies
that may not cover all of the newly edited FCAR rulesOur approach is the
following:

  * •

For each model, we keep the rule/policy match sentences if the cosine
similarity (6) is above $0.7$. This is an empirical choice that provides fair
matches.

  * •

We keep pairs that have been identified by a number of models greater than
$\sqrt{N}$. Given that the standard deviation of shot noise is equal to the
square root of the average number of events N, hence if models are
uncorrelated, selecting a signal to noise match above one sigma should
correspond to a $\sqrt{N}$ cut.

We end up with $1,760$ matching (rule, policy) pairs out of which we keep
$1,408$ pairs to fine-tune our model (Section 4.2) and $352$ for the
validation dataset. In Figure 1 we can see a sample of our validation dataset
results. This dataset will be good to identify pairs that share high
similarities but will not be able to pick up on subtle pairs. This is why it
is not an unbiased dataset for fine-tuning purposes but we proceed anyway due
to the absence of hand labelled pairing between rules and policies.

Figure 1: Sample of the semantic text search output between rules and
policies. The score corresponds to the percentage of models that have
identified the same match.

###  4.2 Domain Adaptation Pre-Training method

The corpus we use for further pre-training (Section 3.1-Step 1) are rules from
the FCA Rulebook. We use a full word mask and we use the
DataCollatorForLanguageModeling which is a class of transformers to perform
the MLM training666see
huggingface.co/docs/transformers/main_classes/data_collator and
www.sbert.net/examples/unsupervised_learning/MLM/README.html. Then we run Step
2 of Section 3.1 with the MNR loss using our training dataset matching (rule,
policy) pairs. We perform Step 1 and Step 2 using both BERT and all-
MiniLM-L6-v2 pre-train model to further train in our domain-specific corpus.

####  4.2.1 Results

In the absence of an unbiased validation dataset, it is difficult to perform
proper model comparison. We do so by rating two scores.

  * •

Score 1: margin between matching and random pairs. For all the rules in our
validation datasets we use the model considered to compute the cosine
similarity $\mathcal{S}_{c}(R_{i},P_{i})$ between rule $i$ and policy $i$ (the
matching pair) and the cosine similarity $\mathcal{S}_{c}(R_{i},P_{j})$
between rule $i$ and a random policy $j$, with $j\neq i$. Then we compute

$\mathcal{S}_{1}:=\frac{1}{N}\sum_{i=1}^{N}\Big{\\{}\mathcal{S}_{c}(R_{i},P_{i})-\mathcal{S}_{c}(R_{i},P_{j})\Big{\\}},$
(9)

where $N$ is the length of the validation dataset (here $N=352$).

  * •

Score 2: For each rule $R_{i}$, we compute the highest similarity scoring
policy
$P_{i}^{*}:=\operatorname*{arg\,max}_{j=1,\ldots,N}\mathcal{S}_{c}(R_{i},P_{j})$.
If $P_{i}^{*}=P_{i}$, we then increment $\mathcal{S}_{2}$ by one and then
divide by $N$ to get the fraction of ‘correct’ matches per model, namely

$\mathcal{S}_{2}:=\frac{1}{N}\sum_{i=1}^{N}1{1}_{\\{P_{i}=\operatorname*{arg\,max}_{j=1,\ldots,N}\mathcal{S}_{c}(R_{i},P_{j})\\}}.$

For a perfectly tuned model and for an unambiguous validation dataset, $S_{1}$
and $\mathcal{S}_{2}$ should be close to $1$ since
$\mathcal{S}_{c}(R_{i},P_{i})$ approaches $1$ and
$\mathcal{S}_{c}(R_{i},P_{j})$ approaches zero. In practice, since we split
rules and policies in sentences, it may happen that the same sentence occurs
in different policies. Therefore a match between $R_{i}$ and $P_{j}$ (with
$j\neq i$) is not necessarily a wrong match. In addition in each rule, some
sentences are quite generic and are addressed by many policies. Our score does
not account for these and is therefore just a benchmark to test the impact of
the domain adaptation and not an actual score of the quality of semantic
matching. Using BERT as the initial pre-trained model our results are shown in
Table 1.

Table 1: Result for the Domain adaptation with BERT and steps defined in Section 3.1 Model | Score 1 | Score 2
---|---|---
Regular BERT | 0.09 | 0.32
BERT \+ Step 1 | 0.10 | 0.33
BERT \+ Step 1 & 2 | 0.32 | 0.33

The key improvement of the Domain Adaptation comes from fine-tuning Step 2
although we know that regular BERT is not really suited to perform semantic
matching. In fact the low Score 1 for regular BERT comes from the fact that
both positive and negative matches are high. Only fine-tuning can help BERT
distinguish the negative pairing. That is why we also tested the all-
MiniLM-L6-v2 model using it as is, performed Step 1 and Step 2, and quote the
improvement of performing an adaptive pre-training method defined as
$\frac{\text{score fine-tune}-\text{score baseline}}{\text{score baseline}}$.

Table 2: Result for the Domain adaptation with all-MiniLM-L6-v2 and steps defined in Section 3.1 Model | Score 1 | Score 2
---|---|---
all-MiniLM-L6-v2 | 0.21 | 0.46
all-MiniLM-L6-v2 \+ Step 1 & 2 | 0.27 | 0.56
Improvement | 29$\%$ | 22$\%$

This shows again the improvement over the baseline. In addition we tested the
impact of the quality of the training/validation dataset by decreasing the
score threshold from $0.7$ down to $0.6$ when generating the
training/validation dataset in Section 4.1 and found that it decreases both
scores by $10\%$. Finally we find that the fraction of masked words in the
Mask Language Modelling does not impact significantly the results. By default
the fraction is set to 0.15 and we tried 0.2 in the BERT Step 1 and did not
find a change in the scoring. Overall, using limited data for Step 1 and Step
2 we find that the key effect comes from the fine-tuning as was previously
stated in [9].

###  4.3 Fully unsupervised training with the GPL method

This time we perform fine tuning solely by running the GPL method on the FCA
Rulebook corpus and train the GPL method on 10,000 and 20,000 triplets. To do
so we follow the steps described in github.com/UKPLab/gpl using the default
pre-trained model MSMARCO. We output samples at each step to check the pseudo-
labelling. On Step 1, to generate queries from a passage of the corpus to get
positive pairs, such as

######  Example 1.

Selected passage: ‘The investor providing the capital may choose not to be
involved in the running of the venture’.

Generated queries:

  * •

how are investors involved in a venture

  * •

who is involved in the running of the venture

  * •

what are the investors of a venture

######  Example 2.

Selected passage: ‘If a circular submitted for approval is amended a copy of
amended drafts must be resubmitted marked to show changes made to conform with
FCA comments and to indicate other changes’.

Generated queries:

  * •

how to amend FCA circular

  * •

what must FCA amendments show

  * •

how to revise circular for approval

Clearly some of these positive pairs can be noisy. A sample of our generated
training datasets containing our triplets of query, positive, negative
passages is displayed in Figure 2.

Figure 2: Sample of the training dataset generated by the GPL approach from
the FCA Rulebook. The Margin is defined in (8). This data is used to fine-tune
the msmarco pre-trained model.

####  4.3.1 Results

Similarly to the scoring analysis in Section 4.2, we see that the GPL fine-
tuning does not impact the scores as much as the adaptative pre-training
method. The scorings themselves are again to be considered as a benchmark from
the no-fine-tuning scoring which translate into an improvement of the ratio
$\frac{\text{score fine-tune}-\text{score baseline}}{\text{score baseline}}$.
As expected, GPL fine-tuning is not as efficient as mapping rules to policies,
most likely because it did not perform the training over a set of policies but
solely on rules. We also find that performing the training on 10,000 or 20,000
triplets does not impact the results by more than a percent.

Table 3: Result for the GPL fine-tuning on the pre-trained MSMARCO model. Model | Score 1 | Score 2
---|---|---
msmarco | 0.14 | 0.79
msmarco \+ GPL | 0.15 | 0.84
Improvement | 7$\%$ | 6$\%$

##  5 Discussion and comparison between GPL and Adaptative pre-training

Figure 3: Sample comparison of pairing based on cosine similarity from our
pseudo validation dataset where we add the best match of the DA and GPL
approaches.

When looking at the validation dataset for the rule and policy match as well
as the policy identifies by the Adaptative pre-training (DA) and the GPL fine-
tuning, there is no clear way to identify which pairing is the more accurate.
In fact they all return meaningful pairings. In Figure 3 we show a sample
where either the DA or the GPL policy match does not agree with the validation
policy. It seems that the validation policy is not always the best match to
address the rule. In addition, not all the rules have been addressed in our
policy sample. Therefore it is not possible to conclude which approach
performs the best matching to our rules. The three approaches we investigate
in this work are:

  * •

Mean ensemble of N pre-trained models selecting policies if they are returned
by at least $\sqrt{N}$ models with cosine similarity $>0.7$. This is our
validation policy defined in Section 4.1.

  * •

Model from an Adaptative pre-training (DA), defined in Section 3.1

  * •

Model from GPL fully unsupervised training from rules corpus in Section 4.3.

It is likely that a combination of these three approaches should provide the
best pairing. Overall it is quite remarkable that even in the absence of a
training dataset, we manage to perform an Adaptative pre-training that
improves the scores. This can be helpful as an alternative or a complement to
the GPL approach.

## Acknowledgements

The authors would like to thank Rohini Gupta and Amit Madhar from Finreg-E
(https://finreg-e.com), without whom this project would not have been
possible. IA and AJ are supported by the Innovate UK Smart Grant ‘Finreg-E /
Natural Language Processing for Financial Regulation’. DG is supported by the
ESRC Grand Union Doctoral Training Partnership Grant and the Oxford Man
Institute of Quantitative Finance.

## References

  *   * Vaswani et al. [2017] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A., Kaiser, L., Polosukhin, I.: Attention is all you need. Advances in NeurIPS 30 (2017)
  * Firsth [1957] Firsth, J.R.: Studies in Linguistic Analysis. Blackwell, Oxford (1957)
  * Deerwester et al. [1990] Deerwester, S., Dumais, S.T., Furnas, G.W., Landauer, T.K., Harshman, R.: Indexing by latent semantic analysis. Journal of the American Society for Information Science 41(6), 391–407 (1990)
  * Lund and Burgess [1996] Lund, K., Burgess, C.: Producing high-dimensional semantic spaces from lexical co-occurrence. Behavior Research Methods, Instruments, & Computers 28(2), 203–208 (1996)
  * Rohde et al. [2006] Rohde, D.L., Gonnerman, L.M., Plaut, D.C.: An improved model of semantic similarity based on lexical co-occurrence. Communications of the ACM 8(627-633), 116 (2006)
  * Mikolov et al. [2013] Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. arXiv:1301.3781 (2013)
  * Pennington et al. [2014] Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word representation. In: Proceedings of the 2014 Conference on Empirical Methods in NLP, pp. 1532–1543 (2014)
  * Devlin et al. [2018] Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding. In: Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, vol. 1, pp. 4171–4186 (2018)
  * Reimers and Gurevych [2019] Reimers, N., Gurevych, I.: Sentence-Bert: Sentence embeddings using Siamese Bert-networks. arXiv:1908.10084 (2019)
  * Rosenfeld [1996] Rosenfeld, R.: A maximum entropy approach to adaptive statistical language modelling. Computer Speech & Language 10, 187–228 (1996)
  * Guthrie et al. [2006] Guthrie, D., Allison, B., Liu, W., Guthrie, L., Wilks, Y.: A closer look at skip-gram modelling. In: Proceedings of the Fifth International Conference on Language Resources and Evaluation, vol. 6, pp. 1222–1225 (2006)
  * Mikolov et al. [2013] Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J.: Distributed representations of words and phrases and their compositionality. Advances in NeurIPS 26 (2013)
  * Phuong and Hutter [2022] Phuong, M., Hutter, M.: Formal algorithms for transformers. arXiv:2207.09238 (2022)
  * Maziarka et al. [2002] Maziarka, L., Danel, T., Mucha, S., Rataj, K., Tabor, J., Jastrzebski, S.: Molecule attention transformer. arXiv:2002.08264 (2002)
  * Dosovitskiy et al. [2010] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16$\times$16 words: Transformers for image recognition at scale. arXiv:2010.11929 (2010)
  * Brown et al. [2020] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., Amodei, D.: Language models are few-shot learners. Advances in NeurIPS 33, 1877–1901 (2020)
  * Schroff et al. [2015] Schroff, F., Kalenichenko, D., Philbin, J.: Facenet: A unified embedding for face recognition and clustering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 815–823 (2015)
  * Beltagy et al. [2019] Beltagy, I., Lo, K., Cohan, A.: Scibert: A pretrained language model for scientific text. In: Proceedings of the 2019 Conference on Empirical Methods in NLP, pp. 3615–3620 (2019)
  * Lee et al. [2020] Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C.H., Kang, J.: BioBert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics 36(4), 1234–1240 (2020)
  * Wang et al. [2021] Wang, K., Reimers, N., Gurevych, I.: Tsdae: Using transformer-based sequential denoising auto-encoder for unsupervised sentence embedding learning. In: Findings of the Association for Computational Linguistics, pp. 671–688 (2021)
  * Henderson et al. [2017] Henderson, M., Al-Rfou, R., Strope, B., Sung, Y.-H., Lukács, L., Guo, R., Kumar, S., Miklos, B., Kurzweil, R.: Efficient natural language response suggestion for smart reply. arXiv:1705.00652 (2017)
  * Wang et al. [2022] Wang, K., Thakur, N., Reimers, N., Gurevych, I.: GPL: Generative pseudo labeling for unsupervised domain adaptation of dense retrieval. In: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2345–2360 (2022)
  * Raffel et al. [2020] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research 21(1), 5485–5551 (2020)
  * Humeau et al. [2019] Humeau, S., Shuster, K., Lachaux, M.-A., Weston, J.: Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring. arXiv:1905.01969 (2019)


# MINDS. The detection of 13CO2 with JWST-MIRI indicates abundant CO2 in a
protoplanetary disk

Sierra L. Grant Max-Planck Institut für Extraterrestrische Physik (MPE),
Giessenbachstr. 1, 85748, Garching, Germany  Ewine F. van Dishoeck  Leiden
Observatory, Leiden University, 2300 RA Leiden, the Netherlands  Max-Planck
Institut für Extraterrestrische Physik (MPE), Giessenbachstr. 1, 85748,
Garching, Germany  Benoît Tabone  Université Paris-Saclay, CNRS, Institut
d’Astrophysique Spatiale, 91405, Orsay, France  Danny Gasman  Institute of
Astronomy, KU Leuven, Celestijnenlaan 200D, 3001 Leuven, Belgium  Thomas
Henning Max-Planck-Institut für Astronomie (MPIA), Königstuhl 17, 69117
Heidelberg, Germany  Inga Kamp  Kapteyn Astronomical Institute,
Rijksuniversiteit Groningen, Postbus 800, 9700AV Groningen, The Netherlands
Manuel Güdel Dept. of Astrophysics, University of Vienna, Türkenschanzstr 17,
A-1180 Vienna, Austria  Max-Planck-Institut für Astronomie (MPIA), Königstuhl
17, 69117 Heidelberg, Germany  ETH Zürich, Institute for Particle Physics and
Astrophysics, Wolfgang-Pauli-Str. 27, 8093 Zürich, Switzerland  Pierre-Olivier
Lagage  Université Paris-Saclay, Université Paris Cité, CEA, CNRS, AIM,
F-91191 Gif-sur-Yvette, France  Giulio Bettoni  Max-Planck Institut für
Extraterrestrische Physik (MPE), Giessenbachstr. 1, 85748, Garching, Germany
Giulia Perotti Max-Planck-Institut für Astronomie (MPIA), Königstuhl 17, 69117
Heidelberg, Germany  Valentin Christiaens  STAR Institute, Université de
Liège, Allée du Six Août 19c, 4000 Liège, Belgium  Matthias Samland  Max-
Planck-Institut für Astronomie (MPIA), Königstuhl 17, 69117 Heidelberg,
Germany  Aditya M. Arabhavi Kapteyn Astronomical Institute, Rijksuniversiteit
Groningen, Postbus 800, 9700AV Groningen, The Netherlands  Ioannis Argyriou
Institute of Astronomy, KU Leuven, Celestijnenlaan 200D, 3001 Leuven, Belgium
Alain Abergel  Université Paris-Saclay, CNRS, Institut d’Astrophysique
Spatiale, 91405, Orsay, France  Olivier Absil STAR Institute, Université de
Liège, Allée du Six Août 19c, 4000 Liège, Belgium  David Barrado Centro de
Astrobiología (CAB), CSIC-INTA, ESAC Campus, Camino Bajo del Castillo s/n,
28692 Villanueva de la Cañada, Madrid, Spain  Anthony Boccaletti  LESIA,
Observatoire de Paris, Université PSL, CNRS, Sorbonne Université, Université
de Paris, 5 place Jules Janssen, 92195 Meudon, France  Jeroen Bouwman Max-
Planck-Institut für Astronomie (MPIA), Königstuhl 17, 69117 Heidelberg,
Germany  Alessio Caratti o Garatti  INAF – Osservatorio Astronomico di
Capodimonte, Salita Moiariello 16, 80131 Napoli, Italy  Dublin Institute for
Advanced Studies, 31 Fitzwilliam Place, D02, XF86 Dublin, Ireland  Vincent
Geers  UK Astronomy Technology Centre, Royal Observatory Edinburgh, Blackford
Hill, Edinburgh EH9 3HJ, UK  Adrian M. Glauser ETH Zürich, Institute for
Particle Physics and Astrophysics, Wolfgang-Pauli-Str. 27, 8093 Zürich,
Switzerland  Rodrigo Guadarrama  Dept. of Astrophysics, University of Vienna,
Türkenschanzstr 17, A-1180 Vienna, Austria  Hyerin Jang  Department of
Astrophysics/IMAPP, Radboud University, PO Box 9010, 6500 GL Nijmegen, The
Netherlands  Jayatee Kanwar  Kapteyn Astronomical Institute, Rijksuniversiteit
Groningen, Postbus 800, 9700AV Groningen, The Netherlands  Space Research
Institute, Austrian Academy of Sciences, Schmiedlstr. 6, A-8042, Graz, Austria
Fred Lahuis  SRON Netherlands Institute for Space Research, PO Box 800, 9700
AV, Groningen, The Netherlands  Maria Morales-Calderón  Centro de
Astrobiología (CAB), CSIC-INTA, ESAC Campus, Camino Bajo del Castillo s/n,
28692 Villanueva de la Cañada, Madrid, Spain  Michael Mueller Kapteyn
Astronomical Institute, Rijksuniversiteit Groningen, Postbus 800, 9700AV
Groningen, The Netherlands  Cyrine Nehmé  Université Paris-Saclay, Université
Paris Cité, CEA, CNRS, AIM, F-91191 Gif-sur-Yvette, France  Göran Olofsson
Department of Astronomy, Stockholm University, AlbaNova University Center,
10691 Stockholm, Sweden  Eric Pantin  Université Paris-Saclay, Université
Paris Cité, CEA, CNRS, AIM, F-91191 Gif-sur-Yvette, France  Nicole Pawellek
Dept. of Astrophysics, University of Vienna, Türkenschanzstr 17, A-1180
Vienna, Austria  Tom P. Ray Dublin Institute for Advanced Studies, 31
Fitzwilliam Place, D02, XF86 Dublin, Ireland  Donna Rodgers-Lee  Dublin
Institute for Advanced Studies, 31 Fitzwilliam Place, D02, XF86 Dublin,
Ireland  Silvia Scheithauer Max-Planck-Institut für Astronomie (MPIA),
Königstuhl 17, 69117 Heidelberg, Germany  Jürgen Schreiber  Max-Planck-
Institut für Astronomie (MPIA), Königstuhl 17, 69117 Heidelberg, Germany
Kamber Schwarz  Max-Planck-Institut für Astronomie (MPIA), Königstuhl 17,
69117 Heidelberg, Germany  Milou Temmink  Leiden Observatory, Leiden
University, 2300 RA Leiden, the Netherlands  Bart Vandenbussche  Institute of
Astronomy, KU Leuven, Celestijnenlaan 200D, 3001 Leuven, Belgium  Marissa
Vlasblom  Leiden Observatory, Leiden University, 2300 RA Leiden, the
Netherlands  L. B. F. M. Waters Department of Astrophysics/IMAPP, Radboud
University, PO Box 9010, 6500 GL Nijmegen, The Netherlands  SRON Netherlands
Institute for Space Research, Niels Bohrweg 4, NL-2333 CA Leiden, the
Netherlands  Gillian Wright  UK Astronomy Technology Centre, Royal Observatory
Edinburgh, Blackford Hill, Edinburgh EH9 3HJ, UK  Luis Colina  Centro de
Astrobiología (CAB, CSIC-INTA), Carretera de Ajalvir, E-28850 Torrejón de
Ardoz, Madrid, Spain  Thomas R. Greve  DTU Space, Technical University of
Denmark. Building 328, Elektrovej, 2800 Kgs. Lyngby, Denmark  Kay Justannont
Chalmers University of Technology, Onsala Space Observatory, 439 92 Onsala,
Sweden  Göran Östlin  Department of Astronomy, Stockholm University, AlbaNova
University Center, 10691 Stockholm, Sweden

###### Abstract

We present JWST-MIRI MRS spectra of the protoplanetary disk around the low-
mass T Tauri star GW Lup from the MIRI mid-INfrared Disk Survey (MINDS) GTO
program. Emission from 12CO2, 13CO2, H2O, HCN, and C2H2 is identified in this
disk with 13CO2 being detected for the first time in a protoplanetary disk. We
characterize the chemical and physical conditions in the inner few au of the
GW Lup disk using these molecules as probes. The spectral resolution of JWST-
MIRI MRS paired with high signal-to-noise data is essential to identify these
species and determine their column densities and temperatures. The
$Q$-branches of these molecules, including those of hot-bands, are
particularly sensitive to temperature and column density. We find that the
12CO2 emission in the GW Lup disk is very strong and is coming from optically
thick emission at a temperature of $\sim$500 K. 13CO2 is optically thinner,
and tracing a larger emitting area than 12CO2 and deeper into the disk based
on a lower temperature of $\sim$200 K. The derived
$N_{\rm{CO_{2}}}$/$N_{\rm{H_{2}O}}$ ratio of 0.25 is relatively high compared
to other targets determined from Spitzer-IRS data. This abundance ratio may be
due to an inner cavity in the gas and dust with a radius in between the H2O
and CO2 snowlines and/or a dust trap at this location keeping water-rich icy
grains from passing the water snowline. This paper demonstrates the unique
ability of JWST to probe inner disk structures and chemistry through weak,
previously unseen molecular features.

protoplanetary disks – stars: pre-main sequence – stars: variables: T Tauri,
Herbig Ae/Be – planets and satellites: formation

##  1 Introduction

The inner 10 au of protoplanetary disks are regions of rich chemistry, with
high temperatures and densities and with the snowlines of H2O and CO2
controlling the gas composition (Pontoppidan et al., 2014). The chemistry in
this region will impact the atmospheric compositions of any exoplanets, the
bulk of which are expected to form in this region (Dawson & Johnson, 2018;
Öberg & Bergin, 2021; Mollière et al., 2022).

Of the several species that emit from the inner $\sim$10 au of disks, CO2 is a
particularly informative tracer of the physical and chemical conditions in
this region. In the interstellar medium, ices are rich in CO2 (abundances of
10-5 with respect to the total gas density; de Graauw et al. 1996; Gibb et al.
2004; Bergin et al. 2005; Pontoppidan et al. 2008; Boogert et al. 2015).
However, the CO2 abundance in disks, derived from both LTE slab models and
full disk non-LTE modeling of Spitzer-IRS observations, is between 10-9 and
10-7 with respect to the total gas density, indicating reprocessing in the
disk (Pontoppidan & Blevins, 2014; Salyk et al., 2011; Bosman et al., 2017).
Despite these lower disk abundances, Pontoppidan et al. (2010) found that CO2
was the second most common molecule detected in disks (20 disks) after water
(25 disks) in a sample of 73 protoplanetary disks observed with Spitzer-IRS.
In those sources, the CO2 $Q$-branch at 14.9 $\mu$m was particularly useful as
a diagnostic of the gas temperature and abundance in their inner regions.
Besides ice production, CO2 is also formed in the gas phase at moderate
temperatures (100 to 200 K) through the reaction of CO + OH $\rightarrow$ CO2
\+ H. At higher temperatures, OH primarily reacts with H2 to form H2O. Thus,
the CO2/H2O ratio is sensitive to the gas temperature.

Despite the usefulness of its typically bright $Q$-branch, 12CO2 is thought to
be largely optically thick in the regions of the disk where it is emitting
(Bosman et al., 2017). Therefore, optically thinner lines and isotologues are
more useful in determining the abundance of CO2, as well as the physical and
chemical conditions in the disk. Moderate spectral resolution and high signal-
to-noise observations are needed to detect the weaker optically thin lines and
isotopologues, which was not possible with Spitzer. JWST-MIRI provides a new
opportunity to study the $Q$-branches of 12CO2 and 13CO2, and the ability to
identify individual $P$\- and $R$-branch lines for 12CO2.

We present JWST-MIRI observations of one of the CO2-bright sources identified
in Spitzer observations: GW Lup (Pontoppidan et al., 2010; Salyk et al., 2011;
Bosman et al., 2017). GW Lup (Sz 71) is an M1.5 star ($T_{\rm{eff}}$=3630 K,
$L_{*}$=0.33 $L_{\odot}$, $M_{*}$=0.37 $M_{\odot}$) in the Lupus I cloud at a
distance of 155 pc (Alcalá et al., 2017; Andrews et al., 2018). This target
was observed as part of the DSHARP survey (Andrews et al., 2018), which found
a very narrow ring of continuum emission at a radius of 85 au in addition to a
centrally-peaked continuum (Dullemond et al., 2018). We re-detect C2H2 and
strong 12CO2 emission in this disk and additionally detect 13CO2, H2O, HCN for
the first time in this source. We fit the 13.6 to 16.3 $\mu$m wavelength range
of the MIRI spectrum with local thermodynamic equilibrium (LTE) slab models to
constrain the abundance and temperature for each species. We discuss our
findings, in particular the detection of 13CO2, which is the first such
detection in a protoplanetary disk, and the column density ratio of CO2 to
H2O.

Figure 1: The JWST-MIRI MRS spectrum for GW Lup. We show all channels and sub-
bands through 4A. Three of the strongest emission features are labeled. The
beginning and end of each sub-band has been trimmed to reduce spurious
features due to the increased noise at the ends of the bands.

##  2 Observations and analysis

###  2.1 Observations and data reduction

GW Lup was observed with the Mid-InfraRed Instrument (MIRI; Rieke et al. 2015;
Wells et al. 2015; Wright et al. 2015, Wright et al. submitted, Argyriou et
al. in prep.) in the Medium Resolution Spectroscopy (MRS) mode on 8 August,
2022. These observations are part of the MIRI mid-INfrared Disk Survey (MINDS)
JWST GTO Program (PID: 1282, PI: T. Henning). A four-point dither was
performed in the positive direction. The total exposure time was 1 hour. All
the JWST data used in this paper can be found in MAST:
http://dx.doi.org/10.17909/aez2-za93 (catalog 10.17909/aez2-za93).

The MIRI MRS observations were processed through all three reduction stages
(Bushouse et al., 2022) using Pipeline version 1.8.4. The reference files were
generated from the observation of the reference A-type star HD 163466.
Additional details on the reference files used can be found in Gasman et al.
(submitted). A single dedicated point source fringe flat and dedicated
spectrophotometric calibration were used in the reduction process. We skip the
outlier rejection step in Spec3, as this produces spurious results due to the
under-sampling of the PSF, causing artefacts in the extracted spectrum.
Finally, a manual extraction of the spectra for each sub-band was performed,
including aperture correction, with an aperture size of 2.5$\lambda$/D. The
correction factors are the same as those presented in Argyriou et al. (in
preparation), which include the contribution of the PSF wings to the estimated
background determined from an annulus around the source. The background
emission from this annulus is subtracted and its value ranges from $\sim$0.001
Jy in Channel 1 to $\sim$0.1 Jy around 22 $\mu$m.

The final GW Lup spectrum is presented in Figure 1. The $\nu_{2}$=1-0 12CO2
$Q$-branch is most prominent, but a large number of weaker lines are detected
as well. A spurious, single-pixel spike at 18.8 $\mu$m has been removed. Below
7.5 $\mu$m, the Spitzer low-resolution spectrum of GW Lup has a $\sim$20%
higher flux. However, above 7.5 $\mu$m, the flux of the JWST-MIRI spectrum is
$\sim$15% higher than the Spitzer-IRS high- and low- resolution spectra of
this target, but the overall shape is very similar. These offsets may be due
to calibration issues and/or variability in this system. This will be
investigated in a future work. In this work, we use the MIRI continuum-
subtracted spectrum for analysis.

Figure 2: The 13 to 17 $\mu$m wavelength range of the GW Lup spectrum, with
the JWST-MIRI data (black) compared to a model (red) composed of emission from
C2H2 (yellow), HCN (orange), H2O (blue), 12CO2 (green), and 13CO2 (purple).
The inset shows a zoom-in of the 13CO2 feature.  Figure 3: Zoom-ins of the 15
$\mu$m wavelength range of GW Lup, with the JWST-MIRI data (black) compared to
a model (red) composed of emission from 12CO2 (green) and 13CO2 (purple). The
continuum and the best-fit models of C2H2, HCN, and H2O, shown in Figure 2,
have been subtracted from the GW Lup spectrum. The top row shows the best
optically thick fit from Figure 2, while the bottom row shows the optically
thin 12CO2 fit from Salyk et al. (2011). The optically thick model reproduces
the 12CO2 hot-band $Q$-branches at 13.9 and 16.2 $\mu$m better than the
optically thin model, as well as the 13CO2 feature at 15.42 $\mu$m, assuming a
12CO2/13CO2 ratio of 68.

###  2.2 Slab Modeling Procedure

The MIRI spectrum is continuum subtracted in the 13 to 17 $\mu$m range by
selecting line-free regions and using a cubic spline interpolation to
determine the continuum level. This continuum is then subtracted from the
observed spectra (see Section A and Figure 5).

We fit the 13 to 17 $\mu$m continuum-subtracted spectrum with local
thermodynamic equilibrium (LTE) slab models. The line profile function is
assumed to be Gaussian with a full width at half maximum of $\Delta$V=4.7 km
s-1 ($\sigma$=2 km s-1) as is done in Salyk et al. (2011). At high optical
depth, the line shape becomes a flat-topped pseudo-Gaussian due to saturation
at the line core. These models allow us to reproduce the data with only three
free parameters: the line of sight column density $N$, the gas temperature
$T$, and the emitting area given by $\pi$$R^{2}$ for a disk of emission with
radius $R$. While we report the emitting area in terms of this emitting
radius, the emission could be coming from a ring with an area equivalent to
$\pi$$R^{2}$. We include emission from C2H2, HCN, H2O, 12CO2, and 13CO2. C2H2
and HCN line transitions are derived from the HITRAN database (Gordon et al.,
2022). All the lines within 4-30 $\mu$m range are selected and are converted
into LAMDA format (van der Tak et al., 2020) for compatibility with our slab
model. The partition sums for these molecules are retrieved from the
TIPS$\\_$2021$\\_$PYTHON package provided by the
database111https://hitran.org/suppl/TIPS/TIPS2021/. The H2O data are taken
from Tennyson et al. (2001). Finally, for 12CO2 and 13CO2, the data are those
compiled by Bosman et al. (2017). We vary the emitting area, as described
below, and compute a synthetic spectrum in Jy, assuming a distance to GW Lup
of 155 pc. The model spectrum is then convolved to a resolving power of 2500
for Channel 3 where the emission features are present (Labiano et al., 2021).
Finally, the convolved model spectrum is resampled to have the same wavelength
grid as the observed spectrum.

For each molecule, a grid of models was run with $N$ from 1014 to 1022 cm-2,
in steps of 0.166 in log10-space, and $T$ from 100 to 1500 K, in steps of 25
K. The emitting area is varied by ranging the radius from 0.01 to 10 au in
steps of 0.03 in log10-space. The best-fit $N$ and $T$ are determined using a
$\chi^{2}$ fit (see Appendix B and Figure 6 for more details) between the
continuum subtracted data and the convolved and resampled model spectrum. For
each $N$ and $T$, the best-fit emitting area is determined by minimizing the
$\chi^{2}$. The $\chi^{2}$ fit is done in spectral windows which are selected
to minimize the contribution of emission from other species while still
containing features that help to constrain the fits (e.g., optically thin
lines and line peaks that are sensitive to temperature). The best-fit model is
found for H2O first. This model is then subtracted from the observed,
continuum-subtracted spectrum. We then fit HCN, subtract that model, and
continue that procedure for C2H2, 12CO2, and then the final fit is done for
13CO2. This is illustrated, with the spectral windows used for the $\chi^{2}$
determinations, in Figure 7 in Appendix B. A fixed 12CO2/13CO2 abundance ratio
of 68 is adopted based on the standard local interstellar medium ratio (Wilson
& Rood, 1994). The $\chi^{2}$ maps are shown in Appendix B (Figure 6).

##  3 Results

The best-fit model is presented in Figure 2 together with the continuum-
subtracted spectrum in the 13.6 to 16.3 $\mu$m range. The best-fit model
parameters are given in Table 1. The $\chi^{2}$ maps show that at low column
densities (below $\sim$1017 to 1018 cm-2, depending on the molecule, see
Figure 6), in the optically thin regime, the column density and emitting
radius are completely degenerate (e.g., Salyk et al. 2011). In the optically
thick regime, the emitting radius can more accurately be determined, although
there is still a degeneracy between temperature and column density. Typical
uncertainties can be read from the $\chi^{2}$ maps where the degeneracies are
also evident. The degeneracy between our three free parameters is reduced by
fitting a combination of optically thick and thin lines.

###  3.1 12CO2 and 13CO2

Our best-fitting 12CO2 model has $N$=1$\times$1018 cm-2, a temperature of 475
K, and an emitting radius of 0.09 au. 12CO2 is well constrained to a
temperature below $\sim$600 K. Using similar models and a similar technique on
Spitzer-IRS data, Salyk et al. (2011) fit the fundamental 12CO2 $Q$-branch at
14.9 $\mu$m and find a temperature of 750 K, a column density of
1.6$\times$1015 cm-2, and an emitting radius of 1.01 au. While this optically
thin model from Salyk et al. (2011) reproduces this $Q$-branch, it does not
reproduce the 12CO2 hot-band $Q$-branches at 13.9 $\mu$m and 16.2 $\mu$m
(1000-0110) as well as the optically thick model does (Figure 3).

For 13CO2, the best-fits are in the optically thin regime, wherein there is a
degeneracy between the column density and emitting radius. Therefore, we adopt
a 12CO2/13CO2 ratio of 68 (Wilson & Rood, 1994), which results in a 13CO2
column density of 1$\times$1016 cm-2 and an emitting radius of 0.46 au. The
13CO2 temperature is lower than that of 12CO2, with the best-fit temperature
of 225 K. This indicates that the optically thin 13CO2 is indeed tracing
deeper layers into the disk. The combination of 13CO2 and the P(23) line of
12CO2, reproduces the feature at 15.42 $\mu$m.

###  3.2 H2O

We include emission from both para- and ortho- water, assuming ortho/para = 3
(e.g., van Dishoeck et al. 2021 and references therein). Many H2O lines are
present in the 13 to 17 $\mu$m region in the GW Lup spectra, however they are
weaker than the main CO2 $Q$-branch and were not seen previously by Spitzer.
An LTE slab model with a temperature of 600 K, a column density of
4$\times$1018 cm-2, and an emitting radius of 0.15 au reproduces the lines in
this region. The fits at high column densities ($\gtrsim$1019 cm-2) tend to be
less well constrained. This similar H2O column density compared to CO2 is in
contrast to the much lower CO2/H2O ratios found in the large T Tauri Spitzer
sample by Salyk et al. (2011), which is discussed in Section 4.

###  3.3 Other species

For C2H2 and HCN (including the 0200-0110 HCN hot-band $Q$-branch at 14.3
$\mu$m), the fits point to optically thin emission, with well constrained
temperatures of $\sim$500 K and $\sim$1000 K, respectively. This high HCN
temperature is needed to reproduce the ratio of line peaks in the main
$Q$-branch. The column densities for C2H2 and HCN are degenerate with the
emitting areas/radii in this optically thin regime, although C2H2 is on the
border between being optically thick and optically thin. The emitting area for
C2H2 and HCN is chosen from the best-fit model, but it is only constrained
such that $R\gtrsim$0.05 au.

Table 1: Best-fit model parameters Species | $N$ | $T$ | $R$
---|---|---|---
| [cm-2] | [K] | [au]
H2O | 4$\times$1018 | 600 | 0.15
HCN | 2$\times$1016 | 950 | 0.25
C2H2 | 1$\times$1017 | 425 | 0.08
12CO2 | 1$\times$1018 | 475 | 0.09
13CO2 | 1$\times$1016 | 225 | 0.46

##  4 Discussion

Due to its lower abundance 13CO2 is more optically thin than 12CO2 and is more
sensitive to the total CO2 column. The observations and best-fit models
support the picture of optically thick 12CO2 at higher temperatures, higher in
the disk atmosphere, and optically thin 13CO2 at cooler temperatures, probing
deeper layers in the disk. Bosman et al. (2017) compare the flux of the 13CO2
$Q$-branch at 15.42 $\mu$m to the neighboring 12CO2 P(25) line at 15.45
$\mu$m, and show that this ratio is sensitive to enhancements of CO2 at its
snowline. In GW Lup, the P(25) line is stronger than the P(23) line,
indicative of a contribution from the 1110-1000 hot-band $Q$-branch of 12CO2
that is only present at high temperatures/column densities and was not seen in
the models of Bosman et al. (2017). Therefore, the P(25) line is not a good
representative of an individual $P$-branch line at these $J$-levels. Instead,
the P(27) line at 15.48 $\mu$m can be used. From the best-fit models, the peak
of the 13CO2 $Q$-branch is at 7 mJy, compared to 3.8 mJy for the P(27) line.
This 13CO2/P(27) line ratio of 1.8 points to a low outer (10-8 with respect to
the total gas density) and high inner (10-6) disk CO2 abundance, potentially
pointing to icy pebble drift. Future modeling work will be able to confirm
this.

As CO2 and H2O are two of the main oxygen carriers in protoplanetary disks,
the relative abundances of these species is informative. While H2O emission is
present in the MIRI MRS spectrum of GW Lup, it is relatively weak compared to
CO2 with an $N_{\rm{CO}_{2}}$/$N_{\rm{H_{2}O}}$ ratio of 0.25. The large T
Tauri Spitzer sample of Salyk et al. (2011) find a median
$N_{\rm{CO}_{2}}$/$N_{\rm{H_{2}O}}$ ratio of 5$\times$10-4. However, the
abundance ratios from Salyk et al. (2011) are largely derived from the low
column density and high temperature (optically thin) regime that we exclude
using the weaker hot-band 12CO2 Q-branches at 13.9 and 16.2 $\mu$m. Both the
CO2 and H2O emission appear to be coming from close to the star, with best-fit
emitting radii of 0.09 and 0.15 au, respectively. These radii are smaller than
the estimated midplane snowline of H2O, which is at $\sim$0.3-0.4 au for the
stellar mass of GW Lup (Mulders et al., 2015).

At high temperatures ($\gtrsim$250 K), OH will react with H2 to form H2O
(Glassgold et al., 2009). Why then is CO2 so abundant relative to H2O at these
temperatures and locations in this disk? We present three scenarios.

  * •

Pebble drift: If dust grains coated in oxygen-rich ices are drifting inward
from the outer disk, the inner disk will be enriched in oxygen (Banzatti et
al., 2020). However, if this ice enrichment is taking place, both CO2 and H2O
should be enriched. From the modeling, with H2O at significantly higher
temperatures than CO2, it is likely that H2O is coming from the upper layers
of the disk and CO2 closer to the midplane, but above the dust $\tau_{15\mu
m}$=1 line. Indeed, Bosman et al. (2022) cite the detection of 13CO2 as
evidence for a high abundance of deep CO2 that only exists if the H2O
abundance is low. The enrichment from ices could only explain the observed
abundances if there were little vertical mixing that would transport oxygen-
rich material to the disk atmosphere. There may be additional H2O and CO2
hidden below the dust $\tau_{15\mu m}$=1 line.

  * •

Photodissociation of H2O: If the water in the disk atmosphere is being
dissociated by UV photons from the star, then the water abundance will
decrease. However, this is likely not playing a large role in the case of GW
Lup, as we do not detect any OH emission lines that would point to a high rate
of H2O dissociation. Additionally, the CO2 should also be photodissociated.

  * •

Inner cavity and/or dust trap: If there is an inner gas and dust cavity in the
disk that extends to between the H2O and CO2 snowlines (estimated at $\sim$0.4
and $>$1 au, respectively), the H2O will be suppressed but the CO2 will still
be abundant in the gas phase. In this case, the H2O will only be present in
the uppermost layers of the disk atmosphere where the icy grains are warm
enough to sublimate water ice and/or where any free volatile oxygen is driven
into H2O. Additionally, a dust trap between the snowlines, either created by
the same mechanism opening the cavity or by other means, could trap water ice-
rich grains from sublimating but allowing for the sublimation and enrichment
of CO2.

Figure 4:  The Spitzer CO2 vs. H2O fluxes in the sample studied in Banzatti et
al. (2020). Downward-facing triangles are those with CO2 not detected above
3$\sigma$, left-ward facing triangles are those with H2O not detected above
3$\sigma$, and open points are those with neither detected above 3$\sigma$.
Blue points are stars with stellar masses between 0.1 and 0.5 $M_{\odot}$, to
be comparable to GW Lup (red) which has a stellar mass of 0.37 $M_{\odot}$. We
use the Spitzer flux for GW Lup to be consistent with the rest of the sources.

Banzatti et al. (2020) (re-)determined molecular line fluxes for H2O, HCN,
C2H2, and CO2 for the Spitzer sample. While GW Lup has a relatively high CO2
flux and relatively low H2O flux compared to other disks, it is not a complete
outlier (Figure 4). In order to detect 13CO2, CO2 needs to be abundant. Bosman
et al. (2022) show that this occurs if water self-shielding is suppressed,
producing more OH which is needed for additional CO2 formation. There are
several disks in the Spitzer survey analyzed by Pontoppidan et al. (2010),
Salyk et al. (2011), and Banzatti et al. (2020) that may fall into this
category: disks with low water fluxes, but high CO2 fluxes such as DN Tau, IM
Lup, MY Lup, and CX Tau (Figure 4). These also may be strong candidates for
inner cavities that could be traced in the dust continuum with near-infrared
interferometry or in the gas kinematics with high-spectral resolution
spectroscopy, for instance, of the CO ro-vibrational lines at 4.7 $\mu$m
(e.g., Brown et al. 2013; Banzatti et al. 2022). With the sensitivity and
resolution of MIRI, there are likely to be other disks that will show 13CO2
emission, which will allow us to derive strong constraints on the CO2
abundance in these disks.

##  5 Summary and Conclusions

  1. 1.

We identify 12CO2, 13CO2, H2O, HCN, and C2H2 in the JWST-MIRI spectrum of GW
Lup. Using LTE slab models, we reproduce the 13.6 to 16.3 $\mu$m spectrum.
H2O, HCN, and 13CO2 are detected for the first time in this disk, as the
features had line/continuum ratios that were too low to be detectable at the
spectral resolution of Spitzer-IRS.

  2. 2.

The gas-phase 13CO2 detection is the first in a protoplanetary disk. This
detection points to a high CO2 abundance deep into the disk, with a 12CO2
column density of 1$\times$1018 cm-2, temperature of 475 K, and an emitting
radius of 0.09 au.

  3. 3.

The column density ratio of CO2 to H2O derived from LTE slab models
($N_{\rm{CO}_{2}}$/$N_{\rm{H_{2}O}}$=0.25) that fit simultaneously the 12CO2
hot-bands, is over two orders of magnitude higher than what has previously
been found in typical T Tauri disks. This may indicate an inner cavity with a
radius in between the H2O and CO2 midplane snowlines in the disk of GW Lup.

  4. 4.

While GW Lup has a relatively high CO2 flux relative to H2O, as seen with
Spitzer, it is not completely an outlier, suggesting that other disks, such as
those around MY Lup, IM Lup, DN Tau, and CX Tau are good candidates for the
detection of 13CO2.

Taken together, this study demonstrates that JWST MIRI-MRS has the ability to
provide new and unique constraints on inner disk physical and chemical
structures.

The following National and International Funding Agencies funded and supported
the MIRI development: NASA; ESA; Belgian Science Policy Office (BELSPO);
Centre Nationale d’Etudes Spatiales (CNES); Danish National Space Centre;
Deutsches Zentrum fur Luft-und Raumfahrt (DLR); Enterprise Ireland; Ministerio
De Economía y Competividad; Netherlands Research School for Astronomy (NOVA);
Netherlands Organisation for Scientific Research (NWO); Science and Technology
Facilities Council; Swiss Space Office; Swedish National Space Agency; and UK
Space Agency. E.v.D. acknowledges support from the ERC grant 101019751 MOLDISK
and the Danish National Research Foundation through the Center of Excellence
“InterCat” (DNRF150). B.T. is a Laureate of the Paris Region fellowship
program (which is supported by the Ile-de-France Region) and has received
funding under the Marie Sklodowska-Curie grant agreement No. 945298. D.G.
would like to thank the Research Foundation Flanders for co-financing the
present research (grant number V435622N). T.H. and K.S. acknowledge support
from the ERC Advanced Grant Origins 83 24 28. I.K., A.M.A., and E.v.D.
acknowledge support from grant TOP-1614.001.751 from the Dutch Research
Council (NWO). I.K. and J.K. acknowledge funding from H2020-MSCA-ITN-2019,
grant no. 860470 (CHAMELEON). G.B. thanks the Deutsche Forschungsgemeinschaft
(DFG) - grant 138 325594231, FOR 2634/2. O.A. and V.C. acknowledge funding
from the Belgian F.R.S.-FNRS. I.A. and D.G. thank the European Space Agency
(ESA) and the Belgian Federal Science Policy Office (BELSPO) for their support
in the framework of the PRODEX Programme. D.B. has been funded by Spanish
MCIN/AEI/10.13039/501100011033 grants PID2019-107061GB-C61 and No.
MDM-2017-0737. A.C.G. has been supported by PRIN-INAF MAIN-STREAM 2017 and
from PRIN-INAF 2019 (STRADE). T.P.R acknowledges support from ERC grant 743029
EASY. D.R.L. acknowledges support from Science Foundation Ireland (grant
number 21/PATH-S/9339). L.C. acknowledges support by grant
PIB2021-127718NB-I00, from the Spanish Ministry of Science and
Innovation/State Agency of Research MCIN/AEI/10.13039/501100011033

##  Appendix A Continuum subtraction

The continuum is determined using a cubic spline interpolation
(scipy.interpolate.interp1d) between selected line-free regions in the
spectrum (Figure 5).

Figure 5: Top: The 13 to 17 $\mu$m wavelength range of our JWST-MIRI MRS data
of GW Lup (black). The continuum points that we select are shown as the red
points and the interpolated continuum is shown as the red line. Bottom: The
continuum subtracted spectrum.

##  Appendix B $\chi^{2}$ procedure and maps

The reduced $\chi^{2}$ maps for H2O, HCN, C2H2, 12CO2, and 13CO2 are shown in
Figure 6. The reduced $\chi^{2}$ is determined using the following formula:

$\chi^{2}=\frac{1}{N}\sum_{i=1}^{N}\frac{(F_{obs,i}-F_{mod,i})^{2}}{\sigma^{2}},$
(B1)

where $N$ is the number of resolution elements in the spectral windows that
the fit is done over and $\sigma$ is the standard deviation in a relatively
line-free region from 15.85 to 15.94 $\mu$m (Figure 7). The contours shown in
Figure 6 are the 1$\sigma$, 2$\sigma$, and 3$\sigma$ levels determined as
$\chi^{2}_{min}$ \+ 1, $\chi^{2}_{min}$ \+ 4, and $\chi^{2}_{min}$ \+ 9,
respectively (Fedele et al., 2013). Any contribution from other species in
this line-rich region of the spectrum increases the overall $\chi^{2}$ value,
although the spectral windows are selected to minimize this contribution.

Figure 6: The $\chi^{2}$ maps for H2O, HCN, C2H2, 12CO2, and 13CO2 (from top
to bottom). The color-scale shows $\chi^{2}_{min}/\chi^{2}$. The red, orange,
and yellow contours correspond to the 1$\sigma$, 2$\sigma$, and 3$\sigma$
levels. The white contours show the emitting radii in au, as given by the
labels. The best-fit model is marked as the black plus. The H2O corresponds to
pH2O, therefore the total H2O column density is 4 times higher. The best-fit
model corresponds to $\chi_{min}^{2}$/$\chi^{2}$ = 1.  Figure 7: The best-fit
model procedure is shown here. The top panel shows the best-fit H2O model
(blue) overlaid on the continuum-subtracted JWST-MIRI spectrum. In the second
panel, the black spectrum is the observed spectrum after subtracting the H2O
model from the first panel. The best-fit HCN model is found using this as the
input spectrum. This process continues down the panels. The spectral windows
used for each species fit are shown as the horizontal bars, with the given
starting and ending points. The representative noise level of the spectrum in
this region, as determined by the standard deviation (0.59 mJy), is given in
the top panel, determined in the relatively line-free region from 15.85 to
15.94 $\mu$m.

## References

  * Alcalá et al. (2017) Alcalá, J. M., Manara, C. F., Natta, A., et al. 2017, A&A, 600, A20
  * Andrews et al. (2018) Andrews, S. M., Huang, J., Pérez, L. M., et al. 2018, ApJ, 869, L41
  * Banzatti et al. (2020) Banzatti, A., Pascucci, I., Bosman, A. D., et al. 2020, ApJ, 903, 124
  * Banzatti et al. (2022) Banzatti, A., Abernathy, K. M., Brittain, S., et al. 2022, AJ, 163, 174
  * Bergin et al. (2005) Bergin, E. A., Melnick, G. J., Gerakines, P. A., Neufeld, D. A., & Whittet, D. C. B. 2005, ApJ, 627, L33
  * Boogert et al. (2015) Boogert, A. C. A., Gerakines, P. A., & Whittet, D. C. B. 2015, ARA&A, 53, 541
  * Bosman et al. (2022) Bosman, A. D., Bergin, E. A., Calahan, J. K., & Duval, S. E. 2022, ApJ, 933, L40
  * Bosman et al. (2017) Bosman, A. D., Bruderer, S., & van Dishoeck, E. F. 2017, A&A, 601, A36
  * Brown et al. (2013) Brown, J. M., Pontoppidan, K. M., van Dishoeck, E. F., et al. 2013, ApJ, 770, 94
  * Bushouse et al. (2022) Bushouse, H., Eisenhamer, J., Dencheva, N., et al. 2022, JWST Calibration Pipeline, Zenodo, v1.7.0, Zenodo, doi:10.5281/zenodo.7038885
  * Dawson & Johnson (2018) Dawson, R. I., & Johnson, J. A. 2018, ARA&A, 56, 175
  * de Graauw et al. (1996) de Graauw, T., Whittet, D. C. B., Gerakines, P. A., et al. 1996, A&A, 315, L345
  * Dullemond et al. (2018) Dullemond, C. P., Birnstiel, T., Huang, J., et al. 2018, ApJ, 869, L46
  * Fedele et al. (2013) Fedele, D., Bruderer, S., van Dishoeck, E. F., et al. 2013, A&A, 559, A77
  * Gibb et al. (2004) Gibb, E. L., Whittet, D. C. B., Boogert, A. C. A., & Tielens, A. G. G. M. 2004, ApJS, 151, 35
  * Glassgold et al. (2009) Glassgold, A. E., Meijerink, R., & Najita, J. R. 2009, ApJ, 701, 142
  * Gordon et al. (2022) Gordon, I. E., Rothman, L. S., Hargreaves, R. J., et al. 2022, J. Quant. Spec. Radiat. Transf., 277, 107949
  * Labiano et al. (2021) Labiano, A., Argyriou, I., Álvarez-Márquez, J., et al. 2021, A&A, 656, A57
  * Mollière et al. (2022) Mollière, P., Molyarova, T., Bitsch, B., et al. 2022, ApJ, 934, 74
  * Mulders et al. (2015) Mulders, G. D., Ciesla, F. J., Min, M., & Pascucci, I. 2015, ApJ, 807, 9
  * Öberg & Bergin (2021) Öberg, K. I., & Bergin, E. A. 2021, Phys. Rep., 893, 1
  * Pontoppidan & Blevins (2014) Pontoppidan, K. M., & Blevins, S. M. 2014, Faraday Discussions, 168, 49
  * Pontoppidan et al. (2014) Pontoppidan, K. M., Salyk, C., Bergin, E. A., et al. 2014, in Protostars and Planets VI, ed. H. Beuther, R. S. Klessen, C. P. Dullemond, & T. Henning, 363
  * Pontoppidan et al. (2010) Pontoppidan, K. M., Salyk, C., Blake, G. A., et al. 2010, ApJ, 720, 887
  * Pontoppidan et al. (2008) Pontoppidan, K. M., Boogert, A. C. A., Fraser, H. J., et al. 2008, ApJ, 678, 1005
  * Rieke et al. (2015) Rieke, G. H., Wright, G. S., Böker, T., et al. 2015, PASP, 127, 584
  * Salyk et al. (2011) Salyk, C., Pontoppidan, K. M., Blake, G. A., Najita, J. R., & Carr, J. S. 2011, ApJ, 731, 130
  * Tennyson et al. (2001) Tennyson, J., Zobov, N. F., Williamson, R., Polyansky, O. L., & Bernath, P. F. 2001, Journal of Physical and Chemical Reference Data, 30, 735
  * van der Tak et al. (2020) van der Tak, F. F. S., Lique, F., Faure, A., Black, J. H., & van Dishoeck, E. F. 2020, Atoms, 8, 15
  * van Dishoeck et al. (2021) van Dishoeck, E. F., Kristensen, L. E., Mottram, J. C., et al. 2021, A&A, 648, A24
  * Wells et al. (2015) Wells, M., Pel, J. W., Glasse, A., et al. 2015, PASP, 127, 646
  * Wilson & Rood (1994) Wilson, T. L., & Rood, R. 1994, ARA&A, 32, 191
  * Wright et al. (2015) Wright, G. S., Wright, D., Goodson, G. B., et al. 2015, PASP, 127, 595