# Generating Related Work

Darsh J Shah Regina Barzilay
Computer Science and Artificial Intelligence Lab, MIT
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Communicating new research ideas involves highlighting similarities and
differences with past work. Authors write fluent, often long sections to
survey the distinction of a new paper with related work. In this work we model
generating related work sections while being cognisant of the motivation
behind citing papers. Our content planning model generates a tree of cited
papers before a surface realization model lexicalizes this skeleton. Our model
outperforms several strong state-of-the-art summarization and multi-document
summarization models on generating related work on an ACL Anthology (AA) based
dataset which we contribute.

##  1 Introduction

An essential component of scientific writing is positioning new research in
the landscape of existing work. Commonly presented in a related work section,
this comparison synthesizes information from multiple papers related to the
current research. While identification of such papers can be partially
automated111urlhttps://www.connectedpapers.com, the related work section
itself is written by a human writer. In this paper, we are proposing an
algorithm that can assist with this task. Figure 1 presents an example where
our model generates a related work section for a new paper.

Writing of the related work section is a type of multi-document summarization.
However, most of the existing summarization approaches operate over input
documents with significant content overlap such as news. These techniques are
not applicable to our task since we aim to highlight specific relations of
each input article to the current discovery. Prior research in scientific
discourse (Teufel and Moens, 2002; Teufel et al., 2006) identified that
reasons for citing papers fall into several argumentative classes such as
reliance on a previous result or gap in existing solutions. Therefore, we can
view the task of related work generation as predicting such reasons and
conveying them in a coherent format.

Figure 1: Two related work sections presented for the paper
https://www.aclweb.org/anthology/2020.coling-main.92/. Part (a) was produced
by the authors and (b) was produced by our model.

We implement this approach in a traditional generation pipeline based on a
content planner and surface realizer. Our content planning model organizes the
positioning and grouping of past papers predicting the reasons for citing
them. The model generates a (depth=2) tree by taking in all available past
papers, predicting the individual reasons for citing them, and producing a
sorting of selected papers, grouping them into respective branches with a
combined motivation for citing each branch. The produced tree forms the
framework to generate readable text. Our surface realization model, which
iteratively generates text for every span, fluently lexicalizes the rationale
behind citing a set of papers in every branch. Such realization is depicted in
Figure 2 where a variety of reasons such as similarity (PSim), methodological
comparisons (CoCoGM) or weakness (Weak) of past work are captured by our model
output.

Our strategy allows for refined control over the segments, and can
appropriately utilize citation categorization annotations available.
Specifically, using distant supervision, we can employ the categorizations to
learn a pairwise classification function between a pair of new and old papers
– necessary for generating motivation cognisant outputs. Furthermore, text
segmentation annotations on related work sections enable training our content
planning and step-wise surface realization models. The approach leaves enough
scope for human intervention from an application perspective to modify the
skeleton of cited papers or the generated related section.

We apply our method for generating related work on an ACL Anthology dataset
that we collect. Every related work section cites multiple scientific studies
for varying reasons, making it extremely relevant for our task. We perform
extensive automatic and human evaluation to compare our method against state-
of-the-art multi-document and query driven summarization techniques. Our
approach receives the highest scores on automatic and subjective evaluation
metrics such as RougeL, BertScore and Relevance. For instance, on RougeL our
approach gets an absolute 5% over the next best baseline. The method also
performs strongly in an update setting, further highlighting its real world
applicability.

##  2 Related Work

Figure 2: Model guided related work produced for the current paper using all
ACL Anthology papers which we cite. Our content planner’s output branches
[{reason},(cited papers)] guide the paragraph generation by surface realizer.

#### Multi-document Summarization:

Approaches in neural sequence-to-sequence learning (Rush et al., 2015; Cheng
and Lapata, 2016; See et al., 2017; Subramanian et al., 2019) for document
summarization have shown promise and have been adapted successfully for multi-
document summarization (Zhang et al., 2018; Lebanoff et al., 2018; Baumel et
al., 2018; Amplayo and Lapata, 2019; Fabbri et al., 2019; Lu et al., 2020).
Trained on large amounts of data, these methods have improved upon traditional
extractive (Carbonell and Goldstein, 1998; Radev and McKeown, 1998; Haghighi
and Vanderwende, 2009) and abstractive approaches (McKeown and Radev, 1995;
Ganesan et al., 2010). The key aspect of typical multi-document summarization
solutions is to capture repetitions and similarities in the multiple input
documents (Barzilay et al., 1999; Fabbri et al., 2019). However, in scientific
writing, the goal is typically to highlight differences and identify
dissimilarities amongst past work. Shah et al. (2021) addresses the
contrastive nature of scientific studies, however, in related work generation,
we are interested in generating an entire section describing past works and
their differences in context of the new idea.

Figure 3: Illustrating the flow of our model. $x$ is our paper and the past
papers are the ones mentioned in Figure 2. Content planning produces a tree
$k$ with four branches with respective citation reasons. Surface realization
takes this tree to produce an output $y$.

#### Query Driven Summarization:

Several tasks such as article generation (Liu et al., 2018), dialogue (Moghe
et al., 2018; Weston et al., 2018; Dinan et al., 2018; Fan et al., 2020),
translation (Gu et al., 2018) and language modeling (Guu et al., 2018;
Khandelwal et al., 2019) can be categorized as query driven generation. Our
work can also be considered as part of this framework, with the new work being
a query. However, we are interested in generating a coherent summary which
highlights the comparative aspects of past works.

#### Rhetorical Structure Theory:

Describes the structure of a document in terms of text spans that form
discourse units and the relations between them (Mann and Thompson, 1988) and
is often used for summarization (Goyal and Eisenstein, 2016; Mabona et al.,
2019). For scientific writing, a flat structure of discourse units rather than
a hierarchy has been observed (Teufel and Moens, 2002). Specifically, for
generating related work we base our citation reasoning annotations on Teufel
et al. (2006) to generate informative text. This usage is similar to recent
aspect-oriented summarization approaches (Kunneman et al., 2018; Frermann and
Klementiev, 2019).

##  3 Method

Our goal is to generate a related work section $y$ for a new research idea
$x$, using $k$ past papers and rationales to cite them. In this section, we
describe our solution, illustrated in Figure 3.

###  3.1 Overview

For each paper in the training corpus, we have the abstract $x$, grouping of
papers to be cited along with the reasons for citing them $k$ and the related
work section to be generated $y$. Related work sections can be quite long, and
in order to model their generation, we break $y$ into segments
$\\{y_{1},y_{2},...,y_{n}\\}$ through crowd-source segmentation. Segmentation
annotations and subsequent motivation categorization of text give us a
grouping for cited papers and reasons for citing them
$k=\\{k_{1},k_{2},...,k_{n}\\}$.

The probability of an output summary $y$ is

$P(y|x)=\sum_{k}P(y|k,x)P(k|x)$ (1)
$P(y|x)=\sum_{k}\prod_{i}P(y_{i}|k_{i},x,y_{i-1})\prod_{j}P(k_{j}|x,k_{j-1})$
(2)

allowing us to break the problem into two modules of content planning
$\prod_{j}P(k_{j}|x,k_{j-1})$ and surface realization
$\prod_{i}P(y_{i}|k_{i},x,y_{i-1})$ using Markov assumptions.

The model can be trained and applied in a setting with the gold set of cited
papers or a setting where the full set of AA corpus is present and a relevant
set must be selected through content planning.

###  3.2 Content Planning

Our content planning model takes a new paper’s abstract $x$, a set of
available papers to cite and produces a grouped ordering and reason for citing
each segment $k=\\{k_{1},k_{2},...,k_{n}\\}$. We model our content planning as
a tree (depth=2) generation task as depicted in the upper half of Figure 3.
From the vocabulary of available past papers, our model generates branches of
similar, relevant past papers. The segmentation annotations collected on the
related work sections, form our supervision for this generation. For the set
of papers to be cited $c=\\{c_{1},c_{2},...c_{m}\\}$ we produce a segmented
realization $k=\\{k_{1},k_{2},...,k_{n}\\}$ where
$k_{i}=(\\{c_{e},...,c_{h}\\},r_{i})$, with $\\{c_{e},...,c_{h}\\}$ the set of
papers cited in a branch and $r_{i}$ the reason for citing them. Table 1
mentions a subset of the reasons used in our approach, reported in Teufel et
al. (2006).

At every step of our tree generation, the content planning model $\mathcal{F}$
decides whether a particular paper to be cited $c_{f}$ should be added to the
current branch (segment) $k_{i}$ or to create a new branch $k_{i+1}$. Each
paper $c_{f}$ is represented by the encoding of its title and abstract
$e(c_{f})$ and the research idea $x$ correspondingly $e(x)$. Tree generation
proceeds by considering all yet to be selected past papers and the current
sub-tree $b$. The probability that the past paper $c_{h}$ will be selected for
inclusion in $b$ is

$\pi_{\theta}(c_{h})=\frac{\exp(\mathcal{F}(b,\hat{x},\hat{c_{h}}))}{\sum_{h}\exp(\mathcal{F}(b,\hat{x},\hat{c_{h}}))}$
(3)

Tree generation continues until all papers are used or a maximum number of
steps is reached. Each paper $c_{f}$ has a reason $r_{f}$ for citing it with
respect to the new abstract $x$, predicted through a pair-wise classifier
$r_{f}=f(x,c_{f})$. After running the tree generation, each branch
$\\{c_{e},...,c_{h}\\}$ gets a combined reason for citing based on an
aggregation function $m$ applied to their individual reasons,
$r_{i}=m(r_{e},...,r_{h})$.

The generated content planning tree is used by the surface planning model to
write a related work section.

###  3.3 Surface Realization

The surface realization model $P(y|x,k)$ performs the critical task of
generating a coherent, informative and concise summary $y$ by taking in a new
paper’s abstract $x$ and the content planning tree $k$ from the previous step
as input.

Specifically, we model the long text generation as a step-wise decoding task
$P(y|x,k)=\prod P(y_{i}|k_{i},x,y_{i-1})$, with a single segment $y_{i}$
lexicalized every step. The model is trained using the segmentation
annotations on the related work dataset. The model takes the abstract of new
research $x$, abstracts from to be cited papers $\\{c_{e},...,c_{h}\\}$, a
token to represent the reason to cite them $r_{i}$ and the text span $y_{i-1}$
produced in the previous step. Here, $k_{i}=(\\{c_{e},...,c_{h}\\},r_{i})$
formulates a multi-document summarization task, controlled by a reason
$r_{i}$. Furthermore, $y_{i-1}$ from the previous step guides the generation
of a continuous summary. The model is implemented using a Transformer (Vaswani
et al., 2017) based encoder decoder model. The various inputs $k_{i}$,
$y_{i-1}$ and $x$ are separated using a special token.

##  4 Dataset

Category | Description
---|---
CoCoXY | Contrast between two cited methods.
PSim | Author’s work and cited work are similar.
Neut | Neutral description of cited work.
Weak | Weakness of cited approach.

Table 1: Subset of reasons for citing a paper as described in (Teufel et al.,
2006).

In this section, we describe the dataset introduced for our related work
generation task.

#### AA:

The ACL Anthology (AA) 2020 corpus contains papers on the study of natural
language processing and computational linguistics. This corpora covers varied
topic areas such as text classification, information extraction, generation,
etc. We use this data dump and the corresponding text of the papers, to create
our dataset of (paper, author list, title, abstract, related work section)
tuples.

#### Paper Title and Citation:

We collect the title of the paper and its list of authors. The publication
year and author list allow us to identify the acronym used to cite the paper
in future works.

#### Abstract:

We collect the abstract for all papers, which form the description $x$ used to
generate related work sections.

#### Related Work Section:

We parse the related work section for several papers in the AA corpus. These
sections contain descriptions of past work, indicated explicitly through
acronyms, highlighting the foundations and novelty of new papers.

#### Parallel Data:

Considering papers with related work sections and available papers cited, we
gather a reasonably sized parallel corpora of 8143 data points split into
training, validation and testing sets. All papers published in or before the
year 2019 are used for training and 185 and 202 papers from the year 2020 are
used for validation and testing respectively. 222Data points considered for
evaluation cite at least 15 past papers.

#### Segmentation Annotations:

In order to model generating related work, we conduct crowd-sourced text
segmentation annotations of the corresponding sections. Annotators are
encouraged to identify atomic segments (one or more complete sentences) which
describe the same set of related facts.

#### Citation Reason Annotations:

Teufel et al. (2006) introduces 26 different reasons for citing past papers
(Table 1). We collect the corpora defined in this paper and train a text
classification model to identify similar reasons on the current ACL Anthology
corpora. Using distant supervision, we use these reasons to collect a pairwise
(new paper, cited paper) citation classification corpus.

##  5 Experiments

In this section, we describe the settings used to study the task, evaluation
metrics, baselines for comparison and implementation details.

#### Settings:

We consider three settings to study the related work generation task.
Known Past Works: We use the gold set of cited papers in order to generate the
related work section.
Full AA Dataset: We consider the full set of ACL Anthology papers and expect
models to cite some relevant ones to generate a related work section.
Related Work Update: We stimulate a related work modification, where an
additional paper is to be cited in an otherwise well written related work
section. This scenario is pervasive when authors miss a few references or a
new paper is published while writing.

#### Evaluation Metrics:

We evaluate our systems using the following automatic metrics.
Rouge is an n-gram based automatic metric used to compare the model output
with the gold reference (Lin, 2004).
BertScore is a contextualized embeddings based automatic metric used to
compare the model output with the gold reference (Zhang et al., 2019).
$k$ Perplexity↔ calculates the perplexity (mean of perplexity and reverse
perplexity $\leftrightarrow$) of the reasoning outputs inferred on the model
outputs in context of those from human written related work sections in the
training data.
SARI is a text update evaluation metric (Xu et al., 2016), comparing the
number of uni-grams added or kept compared to the gold update.

In addition to automatic evaluation, we have human annotators score our models
on Relevance and Fluency for 100 data points per model.
Relevance: Indicates if the generated text shares similar information with the
reference section.
Fluency: Represents if the generated text is grammatically correct and written
in well-formed English.
Annotators rate relevance and fluency on a 1-5 likert scale Albaum (1997). We
have 3 annotators score every data point and report the average across the
scores.

| Automatic Evaluation | Human Scores
---|---|---
Model | RougeL | Bert-Score |  $k$ Perplexity↔ | Relevance | Fluency
Copy-Gen | 0.17 | 0.60 | 19.8 | 3.47 | 3.75
Split-Encoder | 0.19 | 0.59 | 18.8 | 3.50 | 3.66
MultiDocTransformer | 0.14 | 0.52 | 19.8 | 3.54 | 3.66
TransformerBART | 0.30 | 0.65 | 12.7 | 3.42 | 3.53
Ours | 0.32 | 0.66 | 10.5 | 3.65 | 3.69

Table 2: Evaluation on standard related work generation task.

#### Baselines:

In order to demonstrate the effectiveness of our method, we compare it against
several state-of-the-art multi-document summarization methods.
Copy-Gen: See et al. (2017) is a summarization technique which can copy from
the input or generate words and recently achieved best results on multi-
document summarization (Lu et al., 2020).
Split-Encoder: Shah et al. (2019) is a double-encoder decoder method for
summarization, technique used in query driven summarization and citation text
generation tasks (Xing et al., 2020).
MultiDocTransformer: Liu and Lapata (2019) is a Transformer (Vaswani et al.,
2017) implementation for multi-document summarization.
TransformerBART: Lewis et al. (2020) is a pre-trained state-of-the-art
summarization Vaswani et al. (2017) based model.

#### Implementation Details:

Our tree generation model $\mathcal{F}$ is implemented as a feed forward
neural network. We use Lewis et al. (2020) for Surface Realization. We use a
BERT (Devlin et al., 2018) based sentence pair classifier for citation reason
identification which takes in the abstracts of the new paper and the to be
cited paper. In the Full AA Setting models use TfIdf similarity to select the
top-$n$ relevant papers to cite.

##  6 Results

In this section, we report the performance of our model and baselines on the
standard, AA vocabulary and update settings. We also perform case study and
ablation analysis.

#### Standard Setting:

In this setting, the model is aware of all the papers to be cited while
generating the related work. Table 2 reports the results of our model and all
competing baselines in this setting. Our model scores higher on RougeL and
BERT-SCORE than the baselines, generating summaries summaries syntactically
and semantically similar to benchmark outputs. $k$ PERPLEXITY↔ scores capture
realistic variability in reasons for citing papers. Our receives the lowest
perplexity score, highlighting its ability to plan and generate a diverse and
realistic sequences of citation reasons.

The Copy-Gen, Split-Encoder, MultiDocTransformer and TransformerBART methods
focus on the repetitions amongst inputs and fail to bring out detailed
similarities and differences necessary in the related work. Our method, on the
other hand is able to better generate longer sections which are closer to
reference summaries and cognisant of the reason for citing papers.

Model | RougeL 10 | RougeL 22
---|---|---
Copy-Gen | 0.16 | 0.11
Split-Encoder | 0.18 | 0.18
MultiDocTransformer | 0.24 | 0.24
TransformerBART | 0.24 | 0.24
Ours | 0.28 | 0.29

Table 3:  Evaluation of full AA database retrieval task.

Model | SARI | RougeL
---|---|---
Copy-Gen | 0.28 | 0.24
TransformerBART | 0.20 | 0.31
MultiDocBART | 0.15 | 0.21
UpdateTransformerBART | 0.15 | 0.22
Ours | 0.34 | 0.61

Table 4:  Evaluation on related work update task.

Model | Uni-gram | Bi-gram | Tri-gram | Four-gram
---|---|---|---|---
Copy-Gen | 0.81 | 0.58 | 0.43 | 0.30
Split-Encoder | 0.56 | 0.20 | 0.04 | 0.0
MultiDocTransformer | 0.56 | 0.11 | 0.01 | 0.0
TransformerBART | 0.76 | 0.44 | 0.30 | 0.24
Ours | 0.78 | 0.39 | 0.21 | 0.15

Table 5:  Fraction of n-grams copied.

Model | Corresponds to Reason
---|---
No-reason input | 0.63
Standard | 0.65

Table 6:  Reason ablation.

Content Planning | Purity
---|---
K-Means | 0.77
Ours | 0.78

Table 7:  Content planning clustering purity.

#### Human Evaluation:

Table 2 also presents the human evaluation scores of the methods on the
related work generation task. Our method is rated the highest by crowd-workers
on Relevance, confirming the automatic evaluation metrics. Methods typically
produce fluent text. The inter-annotator Kappa agreement (McHugh, 2012) is 80%
and 83% for Relevance and Fluency respectively.

#### Full AA Database Setting:

In this setting, the model is not provided with the gold set of papers to be
cited while generating the related work. Our model is trained to select,
cluster and order papers for content planning. Table 3 reports our model
performance in comparison with baselines from the standard setting. Our tree-
segmented content planning model can find a more appropriate ordering of the
papers making it easier for content planning to generate reasonable summaries.
This allows it to outperform the best baseline by an absolute 5% on RougeL (in
the setting where a maximum of 22 papers are cited). The competing baselines
can not deal with the challenges of a growing input and we don’t see
improvements in their output quality when citing more papers.

#### Update Setting:

In this setting, an incomplete related work summary is provided to the model.
The model must modify the related work (with the data from the already cited
works) using the missing cited paper to generate a complete related work.
Table 4 reports the empirical evaluation of our model compared to baselines,
including a setting specific Update-Transformer method. The flexibility of our
tree-segmented content planning approach allows our model to keep most of the
existing summary while updating only the requisite segments in a fluent manner
to add the missing cited paper. On both SARI and RougeL our model outperforms
the baselines by significant margins – absolute 6% on SARI and 30% on RougeL.

#### Case Study:

Table 8 shows outputs for all the baselines and our model on the paper from
Figure 1.Copy-Gen cites several papers, but doesn’t give an informative
summary of their contributions. Alternatively, Split-Encoder, while generating
a longer section, does not capture the context of the new paper in terms of
related work. MultiDocTransformer unfortunately doesn’t generate a
particularly relevant summary for this input. TransformerBART generates a
paragraph which while being fluent, can’t capture the entire pool of related
work. In contrast, our model generates a fluent related work section, covering
a lot of the relevant work. Our model also gives insight into the task tackled
in the current paper.

#### Analysis:

We perform analysis to further study the Content Planning and Surface
Realization models.

#### Content Planning:

Our content planning model allows for an organization of past papers cited in
the related work generation. The tree-segmentation clusters similar papers and
provides the reason for citing each cluster. Table 7 reports the purity score
of the clusters using the tree generation method which is very comparable to a
K-Means clustering method that can not even order the segments.

We develop a BERT (Devlin et al., 2018) sentence pair classifier to judge the
reason for citing a past paper in context of a new paper. We use Teufel et al.
(2006) citation reasoning annotations as distant supervision to produce
training data for sentence pair classification (abstract pairs in this case),
achieving an accuracy of 85% over 26 different classes.

#### Surface Realization:

Our surface realization model, like baselines, generates a lot of new phrases
as reported in Table 5. Our model also produces motivation cognizant outputs
(Figure 6). The 2% improvement compared to no reason input is very significant
as the non Neut reasons are less frequent in the test set.

#### Text Segmentation:

We develop a BERT (Devlin et al., 2018) text segmentation model using MTurk
annotations collected on 180 related work sections. This leads to 6000 data
points for the binary classification of consecutive sentences into same or
different segments. Our model performs this task at a reasonable accuracy of
71% .

##  7 Discussion

While generating long texts, content control is essential. This is
particularly evident in our study of generating related work which we first
model as a tree generation task. This content planning model forms a strong
skeleton for coherent and citation reason specific generation. Subsequent
lexicalization through the surface realization model produces outputs which
outperform those from state-of-the-art methods. This is confirmed by both
automatic and human evaluation.

#### Human Association in Application:

Separating the solution into planning and generation welcomes human
intervention in the writing process. Our reason classification receives an
accuracy of 85% and content planning grouping purity is 78%. While this
performance is indeed promising, the solution also allows for explicit
guidance from humans that can improve the final output. Text generation
through the surface realization model achieves a state-of-the-art 32% RougeL
F1 score. In a few cases, it may generate phrases which are not factually
valid – prudent human validation is encouraged while using such a system.

A human written related work section, on having to accommodate missing papers,
would need updating. This forms another scenario for our model to be used to
augment an already strong human written summary. Our results in this scenario
(Table 4) are promising.

Figures 4 and 5 from Appendix Appendix illustrate examples of an update and
human intervention respectively.

## Ethical considerations

Communicating research ideas is an important task and requires tremendous
human effort. While models developed in this paper may be used to develop
drafts of related work sections by humans, the primary goal of the work is to
model a challenging query driven summarization task in the context of
contrastive information from multiple inputs. If our models are applied in the
real world, the utmost caution must be applied to ensure no relevant paper is
missed or worse, misrepresented. Such mistakes can have a negative impact to a
particularly sacrosanct task of surveying fellow researchers’ work.

Copy-Gen
---
There has been studied in early work such as (Fe-Fei et al., 2003; Fei-Fei et
al., 2006) and more recent work (Ba et al., 2016; Santoro et al., 2016;
Munkhdalai and Yu, 2017; Ravi and Larochelle, 2017; Ravi and Larochelle, 2016;
Mishra et al., 2017; Sung et al., 2018; Allen et al., 2019). More recent work
can be divided into two types: similarity-based methods (Vinyals et al., 2016;
Snell et al., 2017; Sung et al., 2018) and optimization-based methods (Ravi
and Larochelle, 2016; Munkhdalai and Yu, 2017).

Split-Encoder
---
Approaches based on metric learning attempt to relate new classes to those in
the training data (Vinyals et al., 2016; Snell et al., 2017; Ravi and
Larochelle, 2017; Finn et al., 2017; Finn et al., 2017; Vinyals et al., 2016;
Snell et al., 2017; Geng et al., 2018; Allen et al., 2018; Allen et al., 2018;
Allen et al., 2019). Few-shot text techniques modify the optimization strategy
to provide a model to few-shot classes (Vinyals et al., 2014;
com/irlab/SmartCiteConBansal et al., 2016; Munkhdalai and Yu, 2017; Finn et
al., 2017; Finn et al., 2017; Vinyals et al., 2016; Snell et al., 2017; Geng
et al., 2018; Allen et al., 2018; Allen et al., 2018; Allen et al., 2019). The
pioneering work in few-shot learning is that of few-shot text is that of
informal texts such as discourse-level entailment, or the existence of the
shared task (Verhagen et al., 2003; Miller et al., 2003; Wang et al., 2014).
Few-shot learning techniques modify the optimization strategy to provide a
model that can rapidly adapt to related tasks such as (Fe-Fei (Kim and
Larochelle, 2017; Mishra et al., 2018; Yu et al., 2018), while meta-learning
techniques modify the optimization (Ba and Larochelle, 2017; Miller et al.,
2017).

MultiDocTransformer
---
-based methods have been proposed for NLP tasks, such as NER (Zhang et al., 2016; Nallapineni and Nallum, 2017; NER, 2017). Recently, NER have been used for NER has been proposed, including NER and NLP (Zhou and Nenstette, 2017) and NNs (Zeng et alia and Nivreiter, 2015; Nenkova, 2016) have been applied to NER. In NER tasks, NLP applications such as machine translation (Zampieri et al.g., 2017) (Zadeh et al.(2018) have also been proposed to NMT (Zhao et al, 2017), NER.(2018), NLP community (Zhuber and Nirenburg, 2018; NLP) have shown to NNLP tasks (Zaman and Nadehong, 2018).

TransformerBART
---
There are two major approaches towards FSL: (1) metric-based approach whose
goal is to learn feature extractor that extract and generalize to emerging
classes (Vinyals et al., 2016; Snell et al.(2017) and (2) optimization based
approach that aims to optimize model parameters from few samples (Santoro et
al, 2016; Finn etAl., 2017).In this work, we use the ARSC dataset to study a
simple application of transfer learning approaches to few-shot
classification.We train a single binary classifier to learn all fewshot
classes jointly by prefixing class identifiers to the input text.

Our Method
---
The seminal work on few-shot learning dates back to the early 2000s (Fei-Fei
et al., 2003; Fei- Fei etAl., 2006).In recent years, transfer learning has
been successfully applied to many NLP tasks, including machine translation
(Geng et al, 2019), part-of-speech tagging, and question answering.In this
work, we use the ARSC dataset to study a simple application of transfer
learning approaches to few- shot classification.There are two major approaches
towards FSL: (1) metric-based approach whose goal is to learn feature
extractor that extract and generalize to emerging classes (Vinyals et
al.2016), and (2) optimization based approach that aims to optimize model
parameters from few samples (Santoro et al.(2016), Finn et al (2017), Ravi and
Larochelle (2017) and Mishra (2018).Recently, there has been a surge of
interest in transfer learning in the field of NLP, especially in the context
of machine translation, where transfer learning can be applied to a variety of
tasks, such as sentiment classification (Deng et al., 2019), question
answering (Wang et al., 2020), and machine translation and summarization (Ding
and Chen, 2019; Deng et al., 2020).

Table 8: Sample outputs for all models considered on the paper from Figure 1.
While baselines produce fluent outputs, our model is most appropriate at
reporting past work in context of the current problem.

## References

  * Albaum (1997) Gerald Albaum. 1997.  The likert scale revisited.  _Market Research Society. Journal._ , 39(2):1–21.
  * Amplayo and Lapata (2019) Reinald Kim Amplayo and Mirella Lapata. 2019.  Informative and controllable opinion summarization.  _arXiv preprint arXiv:1909.02322_.
  * Barzilay et al. (1999) Regina Barzilay, Kathleen McKeown, and Michael Elhadad. 1999.  Information fusion in the context of multi-document summarization.  In _ACL_ , pages 550–557.
  * Baumel et al. (2018) Tal Baumel, Matan Eyal, and Michael Elhadad. 2018.  Query focused abstractive summarization: Incorporating query relevance, multi-document coverage, and summary length constraints into seq2seq models.  _arXiv preprint arXiv:1801.07704_.
  * Carbonell and Goldstein (1998) Jaime Carbonell and Jade Goldstein. 1998.  The use of mmr, diversity-based reranking for reordering documents and producing summaries.  In _Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval_ , pages 335–336.
  * Cheng and Lapata (2016) Jianpeng Cheng and Mirella Lapata. 2016.  Neural summarization by extracting sentences and words.  In _ACL_ , pages 484–494, Berlin, Germany. ACL.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Dinan et al. (2018) Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. 2018.  Wizard of wikipedia: Knowledge-powered conversational agents.  _arXiv preprint arXiv:1811.01241_.
  * Fabbri et al. (2019) Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019.  Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1074–1084, Florence, Italy. Association for Computational Linguistics.
  * Fan et al. (2020) Angela Fan, Claire Gardent, Chloe Braud, and Antoine Bordes. 2020.  Augmenting transformers with knn-based composite memory for dialogue.  _arXiv preprint arXiv:2004.12744_.
  * Frermann and Klementiev (2019) Lea Frermann and Alexandre Klementiev. 2019.  Inducing document structure for aspect-based summarization.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6263–6273, Florence, Italy. Association for Computational Linguistics.
  * Ganesan et al. (2010) Kavita Ganesan, ChengXiang Zhai, and Jiawei Han. 2010.  Opinosis: A graph based approach to abstractive summarization of highly redundant opinions.  _Coling 2010_.
  * Goyal and Eisenstein (2016) Naman Goyal and Jacob Eisenstein. 2016.  A joint model of rhetorical discourse structure and summarization.  In _Proceedings of the Workshop on Structured Prediction for NLP_ , pages 25–34, Austin, TX. Association for Computational Linguistics.
  * Gu et al. (2018) Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor OK Li. 2018.  Search engine guided neural machine translation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32.
  * Guu et al. (2018) Kelvin Guu, Tatsunori B Hashimoto, Yonatan Oren, and Percy Liang. 2018.  Generating sentences by editing prototypes.  _Transactions of the Association for Computational Linguistics_ , 6:437–450.
  * Haghighi and Vanderwende (2009) Aria Haghighi and Lucy Vanderwende. 2009.  Exploring content models for multi-document summarization.  In _NAACL HLT_ , pages 362–370.
  * Khandelwal et al. (2019) Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019\.  Generalization through memorization: Nearest neighbor language models.  _arXiv preprint arXiv:1911.00172_.
  * Kunneman et al. (2018) Florian Kunneman, Sander Wubben, Antal van den Bosch, and Emiel Krahmer. 2018.  Aspect-based summarization of pros and cons in unstructured product reviews.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 2219–2229, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Lebanoff et al. (2018) Logan Lebanoff, Kaiqiang Song, and Fei Liu. 2018.  Adapting the neural encoder-decoder framework from single to multi-document summarization.  _arXiv preprint arXiv:1808.06218_.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81.
  * Liu et al. (2018) Peter J. Liu, Mohammad Ahmad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018.  Generating wikipedia by summarizing long sequences.  In _arXiv preprint arXiv:1801.10198_.
  * Liu and Lapata (2019) Yang Liu and Mirella Lapata. 2019.  Hierarchical transformers for multi-document summarization.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5070–5081, Florence, Italy. Association for Computational Linguistics.
  * Lu et al. (2020) Yao Lu, Yue Dong, and Laurent Charlin. 2020.  Multi-XScience: A large-scale dataset for extreme multi-document summarization of scientific articles.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 8068–8074, Online. Association for Computational Linguistics.
  * Mabona et al. (2019) Amandla Mabona, Laura Rimell, Stephen Clark, and Andreas Vlachos. 2019.  Neural generative rhetorical structure parsing.  _arXiv preprint arXiv:1909.11049_.
  * Mann and Thompson (1988) William C Mann and Sandra A Thompson. 1988.  Rhetorical structure theory: Toward a functional theory of text organization.  _Text_ , 8(3):243–281.
  * McHugh (2012) Mary L McHugh. 2012.  Interrater reliability: the kappa statistic.  _Biochemia medica_ , 22(3):276–282.
  * McKeown and Radev (1995) Kathleen McKeown and Dragomir R Radev. 1995.  Generating summaries of multiple news articles.  In _Proceedings of the 18th annual international ACM SIGIR conference on Research and development in information retrieval_ , pages 74–82.
  * Moghe et al. (2018) Nikita Moghe, Siddhartha Arora, Suman Banerjee, and Mitesh M. Khapra. 2018.  Towards exploiting background knowledge for building conversation systems.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2322–2332, Brussels, Belgium. Association for Computational Linguistics.
  * Radev and McKeown (1998) Dragomir R. Radev and Kathleen R. McKeown. 1998.  Generating natural language summaries from multiple on-line sources.  _Computational Linguistics_ , 24(3):469–500.
  * Rush et al. (2015) Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015.  A neural attention model for abstractive sentence summarization.  In _EMNLP_ , pages 379–389, Lisbon, Portugal. ACL.
  * See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017.  Get to the point: Summarization with pointer-generator networks.  In _ACL_ , pages 1073–1083, Vancouver, Canada. ACL.
  * Shah et al. (2019) Darsh J Shah, Tal Schuster, and Regina Barzilay. 2019.  Automatic fact-guided sentence modification.  _arXiv preprint arXiv:1909.13838_.
  * Shah et al. (2021) Darsh J. Shah, Lili Yu, Tao Lei, and R. Barzilay. 2021.  Nutribullets hybrid: Multi-document health summarization.  In _NAACL_.
  * Subramanian et al. (2019) Sandeep Subramanian, Raymond Li, Jonathan Pilault, and Christopher Pal. 2019.  On extractive and abstractive neural document summarization with transformer language models.  _arXiv preprint arXiv:1909.03186_.
  * Teufel and Moens (2002) Simone Teufel and Marc Moens. 2002.  Summarizing scientific articles: experiments with relevance and rhetorical status.  _Computational linguistics_ , 28(4):409–445.
  * Teufel et al. (2006) Simone Teufel, Advaith Siddharthan, and Dan Tidhar. 2006.  Automatic classification of citation function.  In _Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing_ , pages 103–110, Sydney, Australia. Association for Computational Linguistics.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Weston et al. (2018) Jason Weston, Emily Dinan, and Alexander Miller. 2018.  Retrieve and refine: Improved sequence generation models for dialogue.  In _Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI_ , pages 87–92, Brussels, Belgium. Association for Computational Linguistics.
  * Xing et al. (2020) Xinyu Xing, Xiaosheng Fan, and Xiaojun Wan. 2020.  Automatic generation of citation texts in scholarly papers: A pilot study.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6181–6190, Online. Association for Computational Linguistics.
  * Xu et al. (2016) Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch. 2016\.  Optimizing statistical machine translation for text simplification.  _TACL_ , 4:401–415.
  * Zhang et al. (2018) Jianmin Zhang, Jiwei Tan, and Xiaojun Wan. 2018.  Adapting neural single-document summarization model for abstractive multi-document summarization: A pilot study.  In _Proceedings of the 11th International Conference on Natural Language Generation_ , pages 381–390, Tilburg University, The Netherlands. ACL.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019\.  Bertscore: Evaluating text generation with bert.  _arXiv preprint arXiv:1904.09675_.

## Appendix

#### Update Example:

We demonstrate a scenario where a human written related work missed a critical
paper. When provided with the missing paper (Lin, 2004), our model finds an
appropriate branch for the paper and the surface realization completes the
writing (Figure 4).

Figure 4: Model output for a summary update for an ammortized related work
from the paper https://www.aclweb.org/anthology/2020.acl-main.445/.

#### Human Intervention:

The related work generated by our model in Figure 2, we demonstrate how
intervention can allow for a more pleasing content planning. Figure 5 shows
the ease with which human intervention can be used to benefit the generation
task. The initial reasons for the paper are quite valid, but the modifications
allow for a better generation.

Figure 5: Human intervention to produce a more relevant content planning.


# CAMEMBERT: A Mini-Neptunes GCM Intercomparison, Protocol Version 1.0.
A CUISINES Model Intercomparison Project.

Duncan A. Christie Physics and Astronomy, College of Engineering, Mathematics
and Physical Sciences, University of Exeter, Exeter EX4 4QL, UK  Elspeth K. H.
Lee Center for Space and Habitability, University of Bern,
Gesellschaftsstrasse 6, CH-3012 Bern, Switzerland  Hamish Innes Atmospheric,
Oceanic and Planetary Physics, University of Oxford, UK  Pascal A. Noti Center
for Space and Habitability, University of Bern, Gesellschaftsstrasse 6,
CH-3012 Bern, Switzerland  Benjamin Charnay LESIA, Observatoire de Paris,
Université PSL, CNRS, Sorbonne Université, Université de Paris, 5 place Jules
Janssen, 92195 Meudon, France  Thomas J. Fauchez NASA Goddard Space Flight
Center, 8800 Greenbelt Road, Greenbelt, MD 20771, USA  Integrated Space
Science and Technology Institute, Department of Physics, American University,
Washington DC  NASA GSFC Sellers Exoplanet Environments Collaboration  Nathan
J. Mayne Physics and Astronomy, College of Engineering, Mathematics and
Physical Sciences, University of Exeter, Exeter EX4 4QL, UK  Russell Deitrick
School of Earth and Ocean Sciences, University of Victoria, Victoria, British
Columbia, Canada  Feng Ding School of Engineering and Applied Sciences,
Harvard University, Cambridge, MA 02138, USA  Jennifer J. Greco Department of
Physics and Astronomy, University of Toledo, 2801 W. Bancroft Street, Toledo,
OH 43606, USA  Mark Hammond Atmospheric, Oceanic and Planetary Physics,
University of Oxford, UK  Isaac Malsky Department of Astronomy, University of
Michigan, Ann Arbor, MI 48109, USA  Avi Mandell NASA Goddard Space Flight
Center, 8800 Greenbelt Road, Greenbelt, MD 20771, USA  Emily Rauscher
Department of Astronomy, University of Michigan, Ann Arbor, MI 48109, USA
Michael T. Roman School of Physics and Astronomy, University of Leicester,
University Road, Leicester LE1 7RH, UK  Facultad de Ingeniería y Ciencias,
Universidad Adolfo Ibáñez, Av. Diagonal las Torres 2640, Peñalolén, Santiago,
Chile  Denis E. Sergeev Physics and Astronomy, College of Engineering,
Mathematics and Physical Sciences, University of Exeter, Exeter EX4 4QL, UK
Linda Sohl Center for Climate Systems Research, Columbia University, New York,
NY, USA  NASA Goddard Institute for Space Studies, 2880 Broadway, New York, NY
10025, USA  Maria E. Steinrueck Max Planck Institute for Astronomy, D-69117
Heidelberg, Germany  Martin Turbet Laboratoire de Météorologie Dynamique/IPSL,
CNRS, Sorbonne Université, École Normale Supérieure, PSL Research University,
École Polytechnique, 75005 Paris, France  Eric T. Wolf Laboratory for
Atmospheric and Space Physics, Department of Atmospheric and Oceanic Sciences,
University of Colorado, Boulder, Boulder, CO  NASA GSFC Sellers Exoplanet
Environments Collaboration, Greenbelt, MD  NExSS Virtual Planetary Laboratory,
Seattle, WA  Maria Zamyatina Physics and Astronomy, College of Engineering,
Mathematics and Physical Sciences, University of Exeter, Exeter EX4 4QL, UK
Ludmila Carone Centre for Exoplanet Science, School of Physics & Astronomy,
University of St Andrews, North Haugh, St Andrews KY169SS, UK

###### Abstract

With an increased focus on the observing and modelling of mini-Neptunes, there
comes a need to better understand the tools we use to model their atmospheres.
In this paper, we present the protocol for the CAMEMBERT (Comparing
Atmospheric Models of Extrasolar Mini-neptunes Building and Envisioning
Retrievals and Transits) project, an intercomparison of general circulation
models (GCMs) used by the exoplanetary science community to simulate the
atmospheres of mini-Neptunes. We focus on two targets well studied both
observationally and theoretically with planned JWST Cycle 1 observations: the
warm GJ 1214b and the cooler K2-18b. For each target, we consider a
temperature-forced case, a clear sky dual-grey radiative transfer case, and a
clear sky multi band radiative transfer case, covering a range of complexities
and configurations where we know differences exist between GCMs in the
literature. This paper presents all the details necessary to participate in
the intercomparison, with the intention of presenting the results in future
papers. Currently, there are eight GCMs participating (ExoCAM, Exo-FMS, FMS
PCM, Generic PCM, MITgcm, RM-GCM, THOR, and the UM), and membership in the
project remains open. Those interested in participating are invited to contact
the authors.

Mini Neptunes (1063) — Exoplanet Atmospheres (487)

##  1 Introduction

Super-Earths and mini-Neptunes represent a demarcation, albeit a nebulous one,
between the giant planets with their thick atmospheres dominated by hydrogen
and helium and the terrestrial planets with thinner secondary atmospheres
(Lopez & Fortney, 2014). The planets that have retained their hydrogen-
dominated atmospheres are believed to have undergone runaway accretion during
their formation in order to accumulate a thick atmosphere (Pollack et al.,
1996; Lee et al., 2014) but have also been able to retain some or all of that
atmosphere in the presence of irradiative evaporation (Owen & Jackson, 2012).

Simulating these planets with general circulation models (GCMs) presents a
unique set of challenges not necessarily seen in the Earth sciences community.
The primitive equations, which assume hydrostatic balance, a thin atmosphere,
and a constant gravitational acceleration with height, may not yield accurate
results for cases where the thickness of the modelled atmosphere becomes
significant relative to the radius of the planet, limiting their applicability
to planets with thick atmospheres (see Tokano 2013, Tort et al. 2015, and
Mayne et al. 2019 for a discussions related to Venus and Titan, Earth, and
mini-Neptunes, respectively). It has also been argued that simulations of
mini-Neptunes specifically may have extremely long convergence times,
potentially of 50,000 Earth days of model time or more (Wang & Wordsworth,
2020), which raises questions about the accuracy of simulations of only a few
thousand days, as are common in the exoplanetary modelling community (e.g.,
Charnay et al., 2015; Mayne et al., 2017, 2019). These issues motivate a
better understanding of the tools we use to model these planets.

While intercomparison studies have been somewhat common in the Earth sciences
community (see e.g. Eyring et al. 2016; Pincus et al. 2016; Haarsma et al.
2016; Ullrich et al. 2017), it has not been until recently that
intercomparisons have been done with a focus on exoplanetary targets. Although
not an intercomparison of multiple GCMs, Heng et al. (2011) performed a
comparison of the spectral and finite-difference dynamical cores in the
Geophysical Fluid Dynamics Laboratory (GFDL) Princeton Flexible Modelling
System (FMS) using the hot Jupiter HD209458 b as a test case. The first true
intercomparison of GCMs used in exoplanetary science, Polichtchouk et al.
(2014), looked at highly idealised configurations – a steady state jet, a
baroclinic wave, and diabatic forcing – for five GCMs with the intention of
better understanding their respective dynamical cores. Increasing the
complexity, Yang et al. (2019) compared GCMs in the context of terrestrial
planets, specifically focusing on the cases of an Earth-like planet orbiting a
G-star and a tidally-locked planet around an M-star. More recently, the
TRAPPIST-1 Habitable Atmosphere Intercomparison (THAI; Fauchez et al., 2020)
compared GCM models of TRAPPIST-1e, investigating the dynamics (Turbet et al.,
2021; Sergeev et al., 2021) and the synthetic observations (Fauchez et al.,
2021) resulting from the simulations. In the final paper, they propose a “GCM
uncertainty error bar” of $\sim 50\%$ when interpreting transmission spectra
with the uncertainty explained mostly by the cloud differences found between
GCMs. The ability to provide this form of context to synthetic observations
highlights the value of projects like THAI.

Based on the success of the THAI project, we propose here the CAMEMBERT
(Comparing Atmospheric Models of Extrasolar Mini-neptunes Building and
Envisioning Retrievals and Transits) intercomparison of GCMs modelling mini-
Neptunes under the umbrella of the CUISINES (Climates Using Interactive Suites
of Intercomparisons Nested for Exoplanet Studies) framework for
intercomparisons for exoplanets (Sohl et al., in prep.). The broad objectives
of CUISINES are twofold. First, it provides a meta-framework to quantify, and
potentially mitigate, differences between exoplanet model outputs. Second, it
aims to assess how these output differences affect the synthetic observations
that are used to predict the detectability of atmospheric constituents and to
interpret data from ground and space telescopes. With the increased focus on
mini-Neptunes with the launch of TESS (e.g., Trifonov et al., 2019; Lacedelli
et al., 2021; Burt et al., 2021), CHEOPS (e.g., Bonfanti et al., 2021; Leleu
et al., 2021), JWST (e.g., Greene et al., 2017; Bean et al., 2021; Hu &
Damiano, 2021) and in anticipation of increased efforts to model these
planets, we believe the timing is appropriate for an intercomparison of GCMs
modelling mini-Neptunes to provide a foundational understanding of how our
models behave and how our model choices may impact the interpretation of
observations.

In this first paper, we outline the protocol for the CAMEMBERT model
intercomparison project (MIP), providing both the motivations for the test
cases as well as sufficient details to reproduce them, in the hopes that the
results of this intercomparison can be used as a calibration for future GCMs.
While tests of the protocol were run using the UM and Exo-FMS to gauge the
viability of the protocol, the results from these tests are not presented
here, with the intention being to present the results from all participating
GCMs in one or more followup papers. The outline of this paper is as follows:
an overview of the GCMs currently participating in the intercomparison and a
discussion of the target planets are found is Section 2. The protocol and
associated test cases are described in Section 3. The outputs and diagnostics
are found in Section 4 and a final summary and discussion is found in Section
5.

##  2 The Participating GCMs and the Target Planets

With the goal of the intercomparison being to understand the differences
between GCMs, the progression of simulations can be divided up into two
stages: first, a simple investigation of the dynamics and the dynamical cores
at the heart of each of the GCMs. GCM simulations of GJ 1214b, for example,
show up to three zonal jets, with differing amplitudes depending on the study
(Menou, 2012; Kataria et al., 2014; Charnay et al., 2015; Drummond et al.,
2018; Mayne et al., 2019; Wang & Wordsworth, 2020). In understanding the
origins of these differences, it is essential to be able to disentangle
differences between dynamical cores and differences between radiative transfer
schemes. Once this baseline understanding is established, we progress to a
second stage where we compare models with radiative transfer.

Table 1: Participating GCMs  GCM | References | Point of Contact
---|---|---
ExoCAM | Wolf et al. (2022) | Eric T. Wolf
Exo-FMS | Lee et al. (2021) | Elspeth K. H. Lee
FMS PCM | Ding & Wordsworth (2019, 2020) | Feng Ding
Generic PCM | Wordsworth et al. (2011); Forget et al. (in prep.) | Benjamin Charnay
MITgcm | Adcroft et al. (2004); Showman et al. (2009); Komacek et al. (2017) | Maria E. Steinrueck
RM-GCM | Rauscher & Menou (2010, 2012); Malsky et al. (in prep.) | Emily Rauscher
THOR | Mendonça et al. (2016); Deitrick et al. (2020) | Russell Deitrick
The Unified Model (UM) | Mayne et al. (2014, 2019) | Duncan Christie

The GCMs currently participating, based on an expressed interest at the BUFFET
workshop (https://nexss.info/buffet-registration/) and during the protocol
development process, are listed in Table 1. Although these are the
participants at the time of the publication of this protocol, participation
remains open and other teams are welcome to join.

###  2.1 Choice of Targets

As mini-Neptunes encompass a class of planets with a wide range of orbital and
planetary parameters, we opt to focus on a warm, close-in case – GJ 1214b –
and a cooler case – K2-18b. The specific target planets were selected based on
the existence of previous GCM modelling efforts so as to reduce the barriers
to participation as well as the existence of past and planned observations.

GJ 1214b is the first mini-Neptune discovered (Charbonneau et al., 2009) and
is the archetypal warm mini-Neptune. Due to its proximity to its host star, it
is expected to be tidally locked and possess a constant warm dayside and cool
nightside, with the possibility of multiple zonal jets, as has been seen in
GCM models by Menou (2012); Kataria et al. (2014); Charnay et al. (2015);
Drummond et al. (2018); Mayne et al. (2019); Wang & Wordsworth (2020).
Questions about the applicability of the primitive equations in modelling
these objects (Mayne et al., 2019) as well as questions about the exact flow
structure and the convergence timescale (Wang & Wordsworth, 2020) further
motivate a mini-Neptune intercomparison for GJ 1214b. In addition, a JWST
MIRI/LRS phase curve as well as multiple JWST NIRCam transit observations are
planned (Bean et al., 2021; Greene et al., 2017).

The more recently discovered K2-18b (Montet et al., 2015) is a temperate mini-
Neptune in the habitable zone of its host star. Its place in the habitable
zone combined with a possible detection of water vapour in its atmosphere
(Tsiaras et al., 2019; Benneke et al., 2019) makes it a tempting target for
both characterisation and observation, although the interpretation of the
$1.4\,\mathrm{\mu m}$ signal has been disputed (Bézard et al., 2020; Barclay
et al., 2021). Thus far, it has been modelled in 3D by Charnay et al. (2021)
and Innes & Pierrehumbert (2021), and there are planned JWST transit
observations using NIRSpec (Hu & Damiano, 2021; Madhusudhan et al., 2021),
MIRI (Madhusudhan et al., 2021), and NIRISS (Madhusudhan et al., 2021). While
it is unclear whether or not K2-18b is tidally locked (see e.g., Leconte et
al., 2015; Charnay et al., 2021), we assume that it is for simplicity.

Table 2: Input Parameters

| GJ 1214b | K2-18b
---|---|---
Common Planetary Parameters |  |
Mass ($\mathrm{kg}$) | $4.88\times 10^{25}$ | $5.15\times 10^{25}$
Radius ($\mathrm{m}$) | $1.75\times 10^{7}$ | $1.66\times 10^{7}$
Orbital Radius (AU) | 0.01485 | $0.159$
Orbital and Rotation Period (days) | 1.58 | 32.94
Gravity ($\mathrm{m\,s^{-2}}$) | 10.7 | 12.4
Common Stellar Parameters |  |
Mass ($\mathrm{kg}$) | $2.98\times 10^{29}$ | $7.14\times 10^{29}$
Radius ($\mathrm{m}$) | $1.50\times 10^{8}$ | $3.09\times 10^{8}$
$T_{\mathrm{eff}}$ (K) | $3250$ | $3457$
Metallicity [Fe/H] | $0.29$ | $0.123$
log(g) | $5.026$ | $4.8$
Case 1 |  |
$\Delta T_{\mathrm{eq,max}}$ (K) | 600 | 50
Specific gas constant $R$ ($\mathrm{J\,kg^{-1}\,K^{-1}}$) | $3.513\times 10^{3}$ | $1.732\times 10^{3}$
Specific heat capacity $c_{P}$ ($\mathrm{J\,kg^{-1}\,K^{-1}}$) | $1.200\times 10^{4}$ | $6.682\times 10^{3}$
Mean molar mass $\mu$ ($\mathrm{g}\,\mathrm{mol}^{-1}$) | $2.367$ | $4.801$
Case 2 |  |
Shortwave Absorption $\kappa_{\mathrm{sw}}$ ($\mathrm{m^{2}\,kg^{-1}}$) | $1\times 10^{-4}$ | $2\times 10^{-5}$
Longwave Absorption $\kappa_{\mathrm{lw}}$ ($\mathrm{m^{2}\,kg^{-1}}$) | $3\times 10^{-3}$ | $1.4\times 10^{-2}$
Instellation ($\mathrm{W\,m^{-2}}$) | $2.17\times 10^{4}$ | $1.37\times 10^{3}$
$T_{\mathrm{int}}$ (K) | $100$ | $90$
Case 3 |  |
Stellar Spectrum | 3000 K BT-Settl | 3500 K BT-Settl
| with [Fe/H]=0.3 | with [Fe/H] = 0.0
| and log(g) = 5 | and log(g) = 5

Note. — Planetary parameters for GJ 1214b are from Cloutier et al. (2021) and
parameters for K2-18b are from Benneke et al. (2019) and Cloutier et al.
(2019). Stellar parameters are for GJ 1214 and K2-18 are taken from Cloutier
et al. (2021) and Benneke et al. (2017), respectively. To avoid ambiguities in
the mass and radius values used in the intercomparison, the input values are
quoted in units of $\mathrm{kg}$ and $\mathrm{m}$, respectively, with the
conversion being done using the appropriate conversion factors in Prša et al.
(2016).

##  3 Protocol

In this section we outline the simulations associated with the intercomparison
and their motivations. In general, we seek to maintain consistency in
parameters throughout the protocol to facilitate comparison as complexity is
increased. While the various cases were initially envisioned as a progression,
we encourage participants to join the cases their GCMs are capable of
completing regardless of whether or not their GCMs are capable of completing
cases earlier in the series.

For all models, we adopt a lower boundary pressure of $3\times
10^{6}\,\mathrm{Pa}$. While previous studies have studied higher pressures –
Charnay et al. (2015) and Mayne et al. (2019) used $8\times
10^{6}\,\,\mathrm{Pa}$ and $2\times 10^{7}\,\,\mathrm{Pa}$, respectively – to
simplify the test cases and limit differences between the primitive and less
simplified equations of the dynamics for mini-Neptunes (Mayne et al., 2019),
we limit ourselves to $3\times 10^{6}\,\mathrm{Pa}$. We adopt an upper
boundary pressure $p\leq 10\,\mathrm{Pa}$ for codes which employ a pressure-
based vertical grid. For GCMs which use a height-based vertical grid (e.g.,
the UM), we require a domain height sufficient to include pressures of
$10\,\mathrm{Pa}$ throughout the simulations; however, this may require
varying the domain height on a per-case basis. This will also result in those
GCMs using height-based grids having potentially significantly lower pressures
on the nightside upper boundary, which may necessitate modifications to ensure
stability. Following the lead of the THAI project, we do not place specific
requirements on the timestep or grid spacing, instead encouraging participants
to adopt parameters they would commonly use for exoplanet studies as
requirements for stability may differ between GCMs.

For simplicity, we assume friction-free, impermeable boundaries to avoid
complicating the tests with boundary-layer friction or mass-exchange, and heat
exchange is limited to a fixed internal heat flux with effective temperature
$T_{\mathrm{int}}$. We do not, however, exclude forms of dissipation that may
be required for numerical stability (e.g., sponge layers, artificial
viscosity).

As mini-Neptunes may have enhanced atmospheric metallicities relative to solar
(Fortney et al., 2013), we adopt a value of $100\times$ solar for K2-18b and
use parameters and profiles consistent with this throughout the
intercomparison. While there may be similar motivation to use this value for
GJ 1214b, due to the number of previous simulations using solar metallicity
and the differences between simulations already shown in the literature,
specifically in the number and speed of the zonal jets, we adopt the solar
value as it allows us to probe a part of the parameter space where we know
differences between GCMs already occur.

Simulations are run for a fixed number of Earth days instead of specifying a
convergence condition. The chosen simulation lengths do not ensure convergence
in simulations which include the deep atmosphere, this may not be sufficient
time for the deep atmosphere to have converged to a steady state (Mayne et
al., 2017); however, as we are limited to regions with $p\leq 3\times
10^{6}\,\mathrm{Pa}$ we should not be significantly impacted. While Wang &
Wordsworth (2020) have found in their simulations of GJ 1214b that over
integration times of 50,000 Earth days or more their model atmospheres
transition from two off-equatorial jets to a single equatorial jet, including
such long integration times in an intercomparison would likely prove
computationally prohibitive and limit participation. We instead focus
initially on comparing GCMs over shorter timescales, with hopes of extending
the work in the future to look at these longer timescales. For cases 1 and 3,
we run the simulations for 4,000 Earth days. For case 2, however, we run each
simulation for 10,000 Earth days as the dual grey case is the case
investigated by Wang & Wordsworth (2020). Although this does not approach the
long integration times of Wang & Wordsworth (2020), an integration time of
10,000 Earth days may be sufficient to understand differences between GCMs as
Menou (2012) observed the formation of the central equatorial jet in their
7800 Earth day simulation. It may be inevitable, however, that an
understanding of a possible delayed formation of an equatorial jet may have to
wait for a follow-up study with fewer participants investigating longer
simulation times.

Although mini-Neptunes have the potential for clouds, and GJ 1214b in
particular has been shown to have strong signs of clouds or hazes (Kreidberg
et al., 2014), we do not include a cloudy benchmark as a part of the protocol
as currently there are an insufficient number of GCMs capable of
participating. We do hope that followup studies and intercomparisons will be
able to include a cloud component as clouds will undoubtedly represent an
important constituent of future mini-Neptune models.

###  3.1 Initial Conditions

To initialize our simulations, we use one-dimensional pressure-temperature
profiles (see Figure 1) with no initial winds. For K2-18b, we use a profile
generated using Exo-REM from Charnay et al. (2021) and for GJ 1214b we use a
profile generated using ATMO from Drummond et al. (2018). For all cases, we do
not include any initial latitudinal or longitudinal variation. Each of these
profiles, along with the profiles for the chemical abundances needed for Case
3, is publicly available as an ASCII text file in the CAMEMBERT repository
(see Section 4.2).

Figure 1: Initial temperature profiles for K2-18b and GJ 1214b. The solid
lines indicate the initial temperature profiles for each planet while the
shaded regions indicate the range of equilibrium temperatures
$T_{\mathrm{eq}}$ in the temperature-forced case (see Equation 4).

###  3.2 Case 1: Temperature Forcing

The first case investigated as a part of the intercomparison is similar to the
temperature-forced benchmark of Held & Suarez (1994). The motivation is to
compare the robustness of the dynamical cores without also comparing differing
radiative transfer schemes. In this case, we opt to use a Newtonian cooling
prescription where the temperature $T$ is forced to an equilibrium temperature
profile $T_{\mathrm{eq}}$ on a given radiative timescale
$\tau_{\mathrm{rad}}$,

$\frac{dT}{dt}=\frac{T_{\mathrm{eq}}-T}{\tau_{\mathrm{rad}}}\,\,,$ (1)

where $t$ is time. We adopt a radiative timescale $\tau_{\mathrm{rad}}$ for a
hydrogen-dominated atmosphere (Zhang & Showman, 2017)

$\tau_{\mathrm{rad,H_{2}}}(p)=\begin{cases}10^{4}\,\,\mathrm{s}&p\leq
10^{2}\,\mathrm{Pa}\\\
10^{5/2}p^{3/4}\,\,\mathrm{s}&10^{2}\,\mathrm{Pa}<p<10^{6}\,\mathrm{Pa}\\\
10^{7}\,\mathrm{s}&p\geq 10^{6}\,\mathrm{Pa}\\\ \end{cases}\,\,,$ (2)

$\tau_{\mathrm{rad}}(p)=\tau_{\mathrm{rad,H_{2}}}(p)\left(\frac{c_{p}}{7R/2}\right)\frac{2}{\mu}\,,$
(3)

with the specific values of $R$, $c_{p}$ and $\mu$ found in Table 2.

The equilibrium three-dimensional temperature profile $T_{\mathrm{eq}}$ is
generated from the initial temperature profile $T_{0}(p)$ and temperature
difference $\Delta T_{\mathrm{eq}}(p)$ intended to mimic qualitatively the
day-night temperature contrast expected from tidally-locked planet,

$T_{\mathrm{eq}}=\begin{cases}T_{0}(p)+\Delta
T_{\mathrm{eq}}(p)\left(|\cos\phi\cos\lambda|-\frac{1}{2}\right)&\text{dayside}\\\
T_{0}(p)-\frac{1}{2}\Delta T_{\mathrm{eq}}(p)&\text{nightside}\end{cases}\,,$
(4)

where the longitude is $\lambda$ and the latitude is $\phi$. The temperature
difference $\Delta T_{\mathrm{eq}}$ is taken to be $\Delta
T_{\mathrm{eq,max}}$ at $p\leq 10\,\mathrm{Pa}$ and to decrease linearly with
$\log p$ until $p=10^{6}\,\mathrm{Pa}$ where is becomes zero. The range of
equilibrium temperatures at a given pressure is shown in Figure 1. For GJ
1214b, the contrast parameter $\Delta T_{\mathrm{eq,max}}$ is chosen to
maintain consistency with the temperature forcing tests of Mayne et al.
(2019). As published temperature forcing tests do not exist for K2-18b, we
instead look to Charnay et al. (2021) which shows a more modest $\sim
50\,\mathrm{K}$ temperature contrast at the top of the atmosphere for their
$100\times$ solar metallicity atmosphere, motivating the chosen value of
$\Delta T_{\mathrm{eq,max}}$.

We note that the temperature forced case of K2-18b has presented significant
difficulty in terms of numerical stability for the UM and Exo-FMS in tests of
the protocol. Rather than remove it, we retain it as a part of the protocol
with a note of caution to participants.

###  3.3 Case 2: Grey Radiative Transfer

For the initial investigation of the impact of radiative transfer, we employ a
dual band approximation with the shortwave and longwave absorption
coefficients given in Table 2 to compute heating rates, with the previous
forcing scheme no longer included. The values of $\kappa_{\mathrm{sw}}$ and
$\kappa_{\mathrm{lw}}$ have been calculated by fitting the initial profiles to
the analytic profiles in Guillot (2010)111This method results in slightly
different values of $\kappa_{\mathrm{sw}}$ and $\kappa_{\mathrm{lw}}$ for GJ
1214b compared to those used previously in Menou (2012); however, for
consistency in methodology with what is done for K2-18b, we choose to compute
our own values instead of using the values in Menou (2012). . As participating
GCMs may offer different methods to attenuate the incoming stellar
irradiation, simulations in Cases 2 and 3 are to be run using the plane-
parallel approximation. We adopt this intermediate step before transitioning
to non-grey radiative transfer as previous studies show that disagreements
between GCMs may already exist at this point (e.g., Menou, 2012; Wang &
Wordsworth, 2020).

###  3.4 Case 3: Non-Grey Radiative Transfer with Fixed Abundance Profiles

To model atmospheric chemistry, we limit ourselves to H_2/He-dominated
atmospheres with H_2O, CH_4, NH_3, CO and CO_2 as well as H_2 and He
collisionally-induced absorption (CIA) and Rayleigh scattering as opacity
sources. Volume mixing ratios for each species as a function of gas pressure,
taken from the same simulations that generated the initial conditions, are
shown in Figure 2 and are provided as a part of the initial conditions archive
(see Section 3.1). All participating GCMs are to use these abundance profiles,
as this allows GCMs without coupled chemistry solvers to participate.

For the stellar spectra, we use the model stellar spectra from PHOENIX BT-
Settl (Allard et al., 2012) which closest matches the target star (see Table
2). These spectra are made available in the CUISINES repository along with the
other required inputs. The specific linelists, calculation method, and
spectral resolution are left to the individual groups.

Figure 2: Chemical abundance profiles for K2-18b and GJ 1214b to be used in
Case 3. These are publicly available in the CAMEMBERT repository (see Section
4.2).

##  4 Outputs, Diagnostics, and Archiving

In this section we outline the procedures for formatting and archiving data.
The goal is to standardise the output and storage of data as much as possible
to facilitate not only the initial analysis but also future analysis by third
parties.

###  4.1 Outputting and Formatting of Data

In order to provide sufficient data for comparison, we output diagnostic
fields with a 1000 Earth day frequency without any averaging applied to track
the evolution of the simulated atmospheres. For the final 1000 Earth days,
outputs should be every 50 Earth days for the purposes of averaging. For the
comparison of atmospheric dynamics, we require pressure and temperature as
well as eastward (u), northward (v), and vertical (w) velocity fields for all
cases. For cases 2 and 3, we additionally require longwave and shortwave
heating rates as well as the top of atmosphere (TOA) outgoing longwave
radiation (OLR), outgoing shortwave radiation (OSR), and incoming shortwave
radiation (ISR). As the participating GCMs have different dissipation schemes
which may lead to differing results, dissipation rates for all
drag/damping/sponge schemes should be output as well. The required outputs are
summarised in Table 3.

To facilitate the sharing of data and subsequent analysis, we ask that all GCM
outputs be in the netCDF format with data stored in SI units. The metadata
associated with each variable should include a description of the variable and
the associated units, following the netCDF Climate and Forecasting Metadata
Conventions222https://cfconventions.org/ (last access: 2022 June 21). In the
case of vector quantities, the sign conventions should be such that northward,
eastward, and upward all constitute positive directions for their respective
components. Similarly, data should be stored such that increasing grid indices
corresponds to a positive change in direction. Fields should be placed on a
single rectangular latitude-longitude grid, with longitudes beginning at
$\lambda=0\arcdeg$ and the poles located at latitudes $\phi=\pm 90\arcdeg$. As
the protocol considers only tidally-locked cases, the anti-stellar point
should be located at $(\lambda,\phi)=(0\arcdeg,0\arcdeg)$. Data along cyclic
coordinates should not appear more than once within the dataset. Along the
vertical coordinate, a single pressure, potential temperature, or height grid
should be adopted.

As a number of the participating GCMs are capable of outputting scalar
diagnostics with a higher frequency than other outputs, we ask that those GCMs
with this capability provide the total axial angular momentum, kinetic energy,
and maximum values of each velocity component with an output frequency they
would normally adopt. These data can be provided in a text file at the time of
submission. We do not make these data a requirement of the protocol as it
would require additional development by a significant number of participants.
Instead, for those GCMs that do not provide these data separately, these
scalar diagnostics will be derived via post-processing of the provided
outputs.

Table 3: Summary of Required Outputs and Diagnostics

Type of Output | Outputs | Dimensionality
---|---|---
Common Outputs |  |
Atmospheric Profiles | Temperature, Pressure | 3D
| u, v, w Velocity Fields | 3D
Dissipation | Dissipation Rates for any Drag/Damping/Sponge Scheme | 3D
Case 1 Only |  |
Temperature Forcing | Heating Rate (in $\mathrm{K\,s^{-1}}$) | 3D
Cases 2 and 3 Only |  |
Radiation | OLR, OSR, and ISR | 2D
| Shortwave and Longwave Heating Rates (in $\mathrm{K\,s^{-1}}$) | 3D

Note. — All fields are to be output every 1000 Earth days with the output
frequency increased to every 50 Earth days during the final 1000 days of each
simulation. Details can be found in Section 4.1.

###  4.2 The CAMEMBERT Repositories

The data resulting from the simulations and analysis will be uploaded to the
CAMEMBERT permanent repository at
https://ckan.emac.gsfc.nasa.gov/organization/cuisines-camembert by
participating scientists. These data will be made available for public access
upon the publication of the results. Pre-publication access can be requested
by contacting the authors. Inputs described in this protocol and scripts
related to the analysis of data and production of plots for the publications
will be made available on the CAMEMBERT GitHub repository at
https://github.com/projectcuisines/camembert. Inputs will be available
immediately while scripts to reproduce results will be made publicly available
upon the publication of the results.

###  4.3 Simulated Observables

As discussed in Section 2.1, GJ 1214b and K2-18b were chosen in part because
there are planned observations with JWST in Cycle 1 (Greene et al., 2017; Bean
et al., 2021; Hu & Damiano, 2021; Madhusudhan et al., 2021). The permanent
repository will host the results of the analysis of the results and the post-
processed synthetic observations. Consistent with the other intercomparisons
that are a part of CUISINES, we will use the Planetary Spectrum Generator
(PSG; Villanueva et al., 2018, 2022) to simulate JWST spectra for instruments
and modes used in Cycle 1 observations of GJ 1214b and K2-18b, and subsequent
cycles when available, using the atmospheric outputs provided by each GCM for
each of the three cases.

###  4.4 Environmental Impact

While not related to the accuracy of the GCMs, we add as a part of the
protocol the requirement that participating scientists include estimates of
power consumption and CO_2 emissions associated with each production run
included in the intercomparison. We include this requirement not as point of
comparison between GCMs, as the environmental impact will primarily depend on
the methods of energy generation in the local power grids. Instead, we include
this to highlight the environmental impact of supercomputing and to encourage
providers of supercomputing resources to transition to environmentally-
sustainable energy sources. These data will be reported in the first results
paper.

##  5 Summary

In this paper, we have presented the protocol for the CAMEMBERT project which
seeks to compare GCMs used by the exoplanetary science community, with models
of mini-Neptunes being the primary focus. Two benchmarks were chosen – the
warm GJ 1214b and the relatively cooler K2-18b – based on the volume of prior
modelling work and observational potential, and a series of simulations of
increasing complexity are described to calibrate and compare the participating
GCMs, with all of the requisite parameters provided here and in the CUISINES
repository. Membership in CAMEMBERT remains open, and other groups interested
in participating are invited to contact the authors. Collaboration meetings
will be held in 2022 and beyond, and the aim is to present the results of the
intercomparison in one or more follow-up papers. It is hoped that the results
from this intercomparison will provide a strong foundation for follow-up
studies exploring elements not including in this initial protocol such as
chemistry, clouds, and convergence timescales with any model differences due
to these elements being more easily isolated and interpreted. As CUISINES
brings together researchers employing a diverse range of tools and approaches,
we anticipate that it will act to create new collaborations and stimulate
progress in understanding exoplanets. For CAMEMBERT specifically, connections
with MALBEC (Modeling Atmospheric Lines By the Exoplanet Community, an
intercomparison of radiative transfer codes) and PIE (Photochemical model
Intercomparison for Exoplanet science, an intercomparison of 1D photochemistry
codes) will begin in 2022 with the goal of comparing results and better
informing our own models with the insights from the other projects.

## Acknowledgements

CAMEMBERT belongs to the CUISINES meta-framework, a Nexus for Exoplanet System
Science (NExSS) science working group. The testing of the protocol was done
using Met Office Software. The contributions of D. A. Christie, N. J. Mayne,
D. Sergeev, and M. Zamyatina made use of the ISCA High Performance Computing
Service at the University of Exeter and the DiRAC Data Intensive service at
Leicester, operated by the University of Leicester IT Services, which forms
part of the STFC DiRAC HPC Facility (www.dirac.ac.uk). The equipment for DIRAC
was funded by BEIS capital funding via STFC capital grants ST/K000373/1 and
ST/R002363/1 and STFC DiRAC Operations grant ST/R001014/1. DiRAC is part of
the National e-Infrastructure. The work of D. A. Christie, N. J. Mayne, D.
Sergeev, and M. Zamyatina was also partly funded by the Leverhulme Trust
through a research project grant [RPG-2020-82], a Science and Technology
Facilities Council Consolidated Grant [ST/R000395/1] and a a UKRI Future
Leaders Fellowship [grant number MR/T040866/1]. T. J. Fauchez acknowledges
support from the GSFC Sellers Exoplanet Environments Collaboration (SEEC),
which is funded in part by the NASA Planetary Science Divisions Internal
Scientist Funding Model. E. K. H. Lee and P. A. Noti are supported by the SNSF
Ambizione Fellowship grant (#193448). H. Innes is supported by funding from
the European Research Council (ERC) under the European Union’s Horizon 2020
research and innovation programme (Grant agreement No. 740963). The work of B.
Charnay was granted access to the HPC resources of MesoPSL financed by the
Region Ile de France and the project Equip@Meso (reference ANR-10-EQPX-29-01)
of the programme Investissements d’Avenir supervised by the Agence Nationale
pour la Recherche.

## References

  * Adcroft et al. (2004) Adcroft, A., Campin, J.-M., Hill, C., & Marshall, J. 2004, Monthly Weather Review, 132, 2845, doi: 10.1175/MWR2823.1
  * Allard et al. (2012) Allard, F., Homeier, D., & Freytag, B. 2012, Philosophical Transactions of the Royal Society of London Series A, 370, 2765, doi: 10.1098/rsta.2011.0269
  * Barclay et al. (2021) Barclay, T., Kostov, V. B., Colón, K. D., et al. 2021, arXiv e-prints, arXiv:2109.14608.  https://arxiv.org/abs/2109.14608
  * Bean et al. (2021) Bean, J. L., Kempton, E. M. R., Fu, G., et al. 2021, Unlocking the Mysteries of the Archetype Sub-Neptune GJ1214b with a Full-Orbit Phase Curve, JWST Proposal. Cycle 1
  * Benneke et al. (2017) Benneke, B., Werner, M., Petigura, E., et al. 2017, ApJ, 834, 187, doi: 10.3847/1538-4357/834/2/187
  * Benneke et al. (2019) Benneke, B., Wong, I., Piaulet, C., et al. 2019, ApJ, 887, L14, doi: 10.3847/2041-8213/ab59dc
  * Bézard et al. (2020) Bézard, B., Charnay, B., & Blain, D. 2020, arXiv e-prints, arXiv:2011.10424.  https://arxiv.org/abs/2011.10424
  * Bonfanti et al. (2021) Bonfanti, A., Delrez, L., Hooton, M. J., et al. 2021, A&A, 646, A157, doi: 10.1051/0004-6361/202039608
  * Burt et al. (2021) Burt, J. A., Dragomir, D., Mollière, P., et al. 2021, AJ, 162, 87, doi: 10.3847/1538-3881/ac0432
  * Charbonneau et al. (2009) Charbonneau, D., Berta, Z. K., Irwin, J., et al. 2009, Nature, 462, 891, doi: 10.1038/nature08679
  * Charnay et al. (2021) Charnay, B., Blain, D., Bézard, B., et al. 2021, A&A, 646, A171, doi: 10.1051/0004-6361/202039525
  * Charnay et al. (2015) Charnay, B., Meadows, V., Misra, A., Leconte, J., & Arney, G. 2015, ApJ, 813, L1, doi: 10.1088/2041-8205/813/1/L1
  * Cloutier et al. (2021) Cloutier, R., Charbonneau, D., Deming, D., Bonfils, X., & Astudillo-Defru, N. 2021, AJ, 162, 174, doi: 10.3847/1538-3881/ac1584
  * Cloutier et al. (2019) Cloutier, R., Astudillo-Defru, N., Doyon, R., et al. 2019, A&A, 621, A49, doi: 10.1051/0004-6361/201833995
  * Deitrick et al. (2020) Deitrick, R., Mendonça, J. M., Schroffenegger, U., et al. 2020, ApJS, 248, 30, doi: 10.3847/1538-4365/ab930e
  * Ding & Wordsworth (2019) Ding, F., & Wordsworth, R. D. 2019, ApJ, 878, 117, doi: 10.3847/1538-4357/ab204f
  * Ding & Wordsworth (2020) —. 2020, ApJ, 891, L18, doi: 10.3847/2041-8213/ab77d1
  * Drummond et al. (2018) Drummond, B., Mayne, N. J., Baraffe, I., et al. 2018, A&A, 612, A105, doi: 10.1051/0004-6361/201732010
  * Eyring et al. (2016) Eyring, V., Bony, S., Meehl, G. A., et al. 2016, Geoscientific Model Development, 9, 1937, doi: 10.5194/gmd-9-1937-2016
  * Fauchez et al. (2020) Fauchez, T. J., Turbet, M., Wolf, E. T., et al. 2020, Geoscientific Model Development, 13, 707, doi: 10.5194/gmd-13-707-2020
  * Fauchez et al. (2021) Fauchez, T. J., Villanueva, G. L., Sergeev, D. E., et al. 2021, arXiv e-prints, arXiv:2109.11460.  https://arxiv.org/abs/2109.11460
  * Forget et al. (in prep.) Forget, F., et al. in prep.
  * Fortney et al. (2013) Fortney, J. J., Mordasini, C., Nettelmann, N., et al. 2013, ApJ, 775, 80, doi: 10.1088/0004-637X/775/1/80
  * Greene et al. (2017) Greene, T. P., Beatty, T. G., Rieke, M. J., & Schlawin, E. 2017, Transit Spectroscopy of Mature Planets, JWST Proposal. Cycle 1
  * Guillot (2010) Guillot, T. 2010, A&A, 520, A27, doi: 10.1051/0004-6361/200913396
  * Haarsma et al. (2016) Haarsma, R. J., Roberts, M. J., Vidale, P. L., et al. 2016, Geoscientific Model Development, 9, 4185, doi: 10.5194/gmd-9-4185-2016
  * Held & Suarez (1994) Held, I. M., & Suarez, M. J. 1994, Bulletin of the American Meteorological Society, 75, 1825, doi: 10.1175/1520-0477(1994)075<1825:APFTIO>2.0.CO;2
  * Heng et al. (2011) Heng, K., Menou, K., & Phillipps, P. J. 2011, MNRAS, 413, 2380, doi: 10.1111/j.1365-2966.2011.18315.x
  * Hu & Damiano (2021) Hu, R., & Damiano, M. 2021, Deep Characterization of the Atmosphere of a Temperate Sub-Neptune, JWST Proposal. Cycle 1
  * Innes & Pierrehumbert (2021) Innes, H., & Pierrehumbert, R. T. 2021, arXiv e-prints, arXiv:2112.11108.  https://arxiv.org/abs/2112.11108
  * Kataria et al. (2014) Kataria, T., Showman, A. P., Fortney, J. J., Marley, M. S., & Freedman, R. S. 2014, ApJ, 785, 92, doi: 10.1088/0004-637X/785/2/92
  * Komacek et al. (2017) Komacek, T. D., Showman, A. P., & Tan, X. 2017, ApJ, 835, 198, doi: 10.3847/1538-4357/835/2/198
  * Kreidberg et al. (2014) Kreidberg, L., Bean, J. L., Désert, J.-M., et al. 2014, Nature, 505, 69, doi: 10.1038/nature12888
  * Lacedelli et al. (2021) Lacedelli, G., Malavolta, L., Borsato, L., et al. 2021, MNRAS, 501, 4148, doi: 10.1093/mnras/staa3728
  * Leconte et al. (2015) Leconte, J., Wu, H., Menou, K., & Murray, N. 2015, Science, 347, 632, doi: 10.1126/science.1258686
  * Lee et al. (2014) Lee, E. J., Chiang, E., & Ormel, C. W. 2014, ApJ, 797, 95, doi: 10.1088/0004-637X/797/2/95
  * Lee et al. (2021) Lee, E. K. H., Parmentier, V., Hammond, M., et al. 2021, MNRAS, 506, 2695, doi: 10.1093/mnras/stab1851
  * Leleu et al. (2021) Leleu, A., Alibert, Y., Hara, N. C., et al. 2021, A&A, 649, A26, doi: 10.1051/0004-6361/202039767
  * Lopez & Fortney (2014) Lopez, E. D., & Fortney, J. J. 2014, ApJ, 792, 1, doi: 10.1088/0004-637X/792/1/1
  * Madhusudhan et al. (2021) Madhusudhan, N., Constantinou, S., Moses, J. I., Piette, A., & Sarkar, S. 2021, Chemical Disequilibrium in a Temperate sub-Neptune, JWST Proposal. Cycle 1
  * Malsky et al. (in prep.) Malsky, I., et al. in prep.
  * Mayne et al. (2014) Mayne, N. J., Baraffe, I., Acreman, D. M., et al. 2014, Geoscientific Model Development, 7, 3059, doi: 10.5194/gmd-7-3059-2014
  * Mayne et al. (2019) Mayne, N. J., Drummond, B., Debras, F., et al. 2019, ApJ, 871, 56, doi: 10.3847/1538-4357/aaf6e9
  * Mayne et al. (2017) Mayne, N. J., Debras, F., Baraffe, I., et al. 2017, A&A, 604, A79, doi: 10.1051/0004-6361/201730465
  * Mendonça et al. (2016) Mendonça, J. M., Grimm, S. L., Grosheintz, L., & Heng, K. 2016, ApJ, 829, 115, doi: 10.3847/0004-637X/829/2/115
  * Menou (2012) Menou, K. 2012, ApJ, 744, L16, doi: 10.1088/2041-8205/744/1/L16
  * Montet et al. (2015) Montet, B. T., Morton, T. D., Foreman-Mackey, D., et al. 2015, ApJ, 809, 25, doi: 10.1088/0004-637X/809/1/25
  * Owen & Jackson (2012) Owen, J. E., & Jackson, A. P. 2012, MNRAS, 425, 2931, doi: 10.1111/j.1365-2966.2012.21481.x
  * Pincus et al. (2016) Pincus, R., Forster, P. M., & Stevens, B. 2016, Geoscientific Model Development, 9, 3447, doi: 10.5194/gmd-9-3447-2016
  * Polichtchouk et al. (2014) Polichtchouk, I., Cho, J. Y. K., Watkins, C., et al. 2014, Icarus, 229, 355, doi: 10.1016/j.icarus.2013.11.027
  * Pollack et al. (1996) Pollack, J. B., Hubickyj, O., Bodenheimer, P., et al. 1996, Icarus, 124, 62, doi: 10.1006/icar.1996.0190
  * Prša et al. (2016) Prša, A., Harmanec, P., Torres, G., et al. 2016, AJ, 152, 41, doi: 10.3847/0004-6256/152/2/41
  * Rauscher & Menou (2010) Rauscher, E., & Menou, K. 2010, ApJ, 714, 1334, doi: 10.1088/0004-637X/714/2/1334
  * Rauscher & Menou (2012) —. 2012, ApJ, 750, 96, doi: 10.1088/0004-637X/750/2/96
  * Sergeev et al. (2021) Sergeev, D. E., Fauchez, T. J., Turbet, M., et al. 2021, arXiv e-prints, arXiv:2109.11459.  https://arxiv.org/abs/2109.11459
  * Showman et al. (2009) Showman, A. P., Fortney, J. J., Lian, Y., et al. 2009, ApJ, 699, 564, doi: 10.1088/0004-637X/699/1/564
  * Sohl et al. (in prep.) Sohl, L. E., et al. in prep., The Planetary Science Journal
  * Tokano (2013) Tokano, T. 2013, Geophys. Res. Lett., 40, 4538, doi: 10.1002/grl.50841
  * Tort et al. (2015) Tort, M., Dubos, T., & Melvin, T. 2015, Quarterly Journal of the Royal Meteorological Society, 141, 3056, doi: 10.1002/qj.2590
  * Trifonov et al. (2019) Trifonov, T., Rybizki, J., & Kürster, M. 2019, A&A, 622, L7, doi: 10.1051/0004-6361/201834817
  * Tsiaras et al. (2019) Tsiaras, A., Waldmann, I. P., Tinetti, G., Tennyson, J., & Yurchenko, S. N. 2019, Nature Astronomy, 3, 1086, doi: 10.1038/s41550-019-0878-9
  * Turbet et al. (2021) Turbet, M., Fauchez, T. J., Sergeev, D. E., et al. 2021, arXiv e-prints, arXiv:2109.11457.  https://arxiv.org/abs/2109.11457
  * Ullrich et al. (2017) Ullrich, P. A., Jablonowski, C., Kent, J., et al. 2017, Geoscientific Model Development, 10, 4477, doi: 10.5194/gmd-10-4477-2017
  * Villanueva et al. (2022) Villanueva, G. L., Liuzzi, G., Faggi, S., et al. 2022, Fundamentals of the Planetary Spectrum Generator
  * Villanueva et al. (2018) Villanueva, G. L., Smith, M. D., Protopapa, S., Faggi, S., & Mandell, A. M. 2018, J. Quant. Spec. Radiat. Transf., 217, 86, doi: 10.1016/j.jqsrt.2018.05.023
  * Wang & Wordsworth (2020) Wang, H., & Wordsworth, R. 2020, ApJ, 891, 7, doi: 10.3847/1538-4357/ab6dcc
  * Wolf et al. (2022) Wolf, E. T., Kopparapu, R., Haqq-Misra, J., & Fauchez, T. J. 2022, , 3, 7, doi: 10.3847/PSJ/ac3f3d
  * Wordsworth et al. (2011) Wordsworth, R. D., Forget, F., Selsis, F., et al. 2011, ApJ, 733, L48, doi: 10.1088/2041-8205/733/2/L48
  * Yang et al. (2019) Yang, J., Leconte, J., Wolf, E. T., et al. 2019, ApJ, 875, 46, doi: 10.3847/1538-4357/ab09f1
  * Zhang & Showman (2017) Zhang, X., & Showman, A. P. 2017, ApJ, 836, 73, doi: 10.3847/1538-4357/836/1/73