# NLLG Quarterly arXiv Report 06/23:
What are the most influential current AI Papers?

Steffen Eger, Christoph Leiter, Jonas Belouadi
Ran Zhang, Aida Kostikova, Daniil Larionov, Yanran Chen, Vivian Fresen
Natural Language Learning Group (NLLG), https://nl2g.github.io/

###### Abstract

The rapid growth of information in the field of Generative Artificial
Intelligence (AI), particularly in the subfields of Natural Language
Processing (NLP) and Machine Learning (ML), presents a significant challenge
for researchers and practitioners to keep pace with the latest developments.
To address the problem of information overload, this report by the Natural
Language Learning Group at Bielefeld University focuses on identifying the
most popular papers on arXiv, with a specific emphasis on NLP and ML. The
objective is to offer a quick guide to the most relevant and widely discussed
research, aiding both newcomers and established researchers in staying abreast
of current trends. In particular, we compile a list of the 40 most popular
papers based on normalized citation counts from the first half of 2023. We
observe the dominance of papers related to Large Language Models (LLMs) and
specifically ChatGPT during the first half of 2023, with the latter showing
signs of declining popularity more recently, however. Further, NLP related
papers are the most influential (around 60% of top papers) even though there
are twice as many ML related papers in our data. Core issues investigated in
the most heavily cited papers are: LLM efficiency, evaluation techniques,
ethical considerations, embodied agents, and problem-solving with LLMs.
Additionally, we examine the characteristics of top papers in comparison to
others outside the top-40 list (noticing the top paper’s focus on LLM related
issues and higher number of co-authors) and analyze the citation distributions
in our dataset, among others.

##  1 Introduction

In an era of ever-accelerating information flow, staying abreast of the
overwhelming flood of data and research output is an intimidating task. This
holds true especially in the context of the current large public interest (and
even hype) surrounding Generative AI, with papers disseminated in ever shorter
time intervals. This report, published by the Natural Language Learning Group
(https://nl2g.github.io/) at Bielefeld University, aims to alleviate the
information overload problem, even if only by a small extent, by identifying
the currently most popular papers on the arXiv (https://arxiv.org/),
especially focusing on the AI subfields natural language processing (NLP) and
machine learning (ML) as some of the most vividly discussed research areas,
including in mainstream media. Our intention is to give practitioners,
incomers, and users of AI, from related and non-related fields (e.g., the
social sciences or digital humanities) a quick guide on the most popular and
presumably most relevant papers in order to better and (more) quickly grasp
current developments.

We place particular emphasis on exploring arXiv,111Our report is similar to a
‘conference report’ as a popular form of science communication, e.g.,
https://www.romanklinger.de/blog-assets/2023-05-12/eacl2023-conf-report.pdf.
But instead of focusing on conferences, we focus on arXiv for multiple
reasons: among others, (i) in an age of rapid developments, conferences and
journals are too slow and often lagging behind recent developments; (ii) as
everyone who regularly submits to NLP/ML conferences knows, conferences also
suffer from low reviewing quality, with junior and non-expert reviewers
abounding. Instead, we focus on citations (even though these are not
unproblematic themselves) as a form of large-scale crowd voting.  given its
status as a comprehensive and extremely popular pre-print repository. Notably,
arXiv’s expedited publication process provides a distinct advantage over
traditional conferences and journals, ensuring that the latest research
becomes readily available to the scientific community at a much faster pace.

This report is structured as follows. In Section 2, we outline our
methodology, which is entirely straightforward: we select papers from arXiv
from the first half of the year 2023 and sort them by normalized citation
counts. In Section 3, we show and discuss the list of the 40 most popular
papers — in terms of normalized citation counts — from our arXiv dataset. In
Section 4, we provide an analysis of our arXiv dataset relating to citation
distributions, arXiv categories involved, characteristics of top papers, and
popularity of ‘hype’ concepts such as ChatGPT and large language models (LLMs)
over time. In Section 5, we conclude.

Among our key findings are that: (i) NLP, once a niche area of research, is
now considerably more influential than ML in terms of the citations it
attracts: even though there are twice as many ML papers in our datasets,
$\sim$60% of the most highly cited papers are from NLP; (ii) LLM and ChatGPT
related papers have clearly dominated the first half of 2023, but especially
ChatGPT is now on the decline; (iii) the efficient open-source model LLaMA
from Meta AI is the relatively and absolutely most cited paper in our dataset,
leaving behind the larger and properietary ChatGPT and GPT-4.

Our code and data is available from https://github.com/NL2G/Quaterly-Arxiv.

##  2 Methodology

To identify the most influential papers from the AI subfields NLP and ML, we
used the following methodology.

Dataset name | Size | Time period | # Primary Categories
---|---|---|---
arxiv-0623 | 20,843 | 01/01/2023-06/31/2023 | 123
arxiv-0623-top40 | 40 | 01/01/2023-06/31/2023 | 5
Table 1: Elementary statistics on our two released datasets. Size is the
number of papers in each dataset; the last column gives the number of distinct
primary arXiv categories our papers are assigned to.

  1. 1.

Data Retrieval from arXiv: We collect all papers from 01/01/2023 to 06/31/2023
belonging to the arXiv categories cs.CL (computation and language) and cs.LG
(machine learning) using a Python arXiv API.222ArXiv papers may belong to
several categories. We only require that one of the involved categories be one
of the two indicated. 333We did not include cs.AI (without cs.LG or cs.CL) but
we note that our top-40 list would have looked very similar with our without
the cs.AI requirement. In particular, all top-40 papers would have remained
the same — also note that many cs.AI papers are still included in our dataset,
see below. Our retrieval time is July, 29, 2023 (which is important, because
citation counts are constantly in flux). ArXiv papers can be updated anytime;
we take the date of the first submission of a paper to arXiv as its
publication date.

  2. 2.

z-score calculation: For each paper, we extract its citation count, as a
measure of popularity and arguably importance [1], from Semantic Scholar
https://www.semanticscholar.org/. Since papers published at different time
points may naturally have different citation counts (e.g., older papers have
higher chance of being cited than very novel papers), we calculate a
_normalized citation count_ by determining _how many standard deviations a
paper is above the mean of citations of all papers published in the same week
(Sunday-Saturday)_. This is the so-called z-score of Newman [23]:

$\displaystyle
z_{t}=\frac{c_{t}-\textit{mean}(\mathbf{c}(t))}{\textit{std}(\mathbf{c}(t))}$

for a paper published in week $t$ with citation count $c_{t}$; $\mathbf{c}(t)$
is the list of citation counts of all papers published in week $t$. If a paper
lies several standard deviations above the mean (for all papers published in
the same week), it can be considered excellent for its class. For example, in
a normal distribution, only about 16% of data points lie one standard
deviation above the mean value. As will be seen below, our top papers lie at
least 9–12 standard deviations above the mean.444Our approach of identifying
top papers in arXiv via the zscore is similar to [11].

  3. 3.

Manual Evaluation The published date on arXiv might differ from the actual
first publication/release/submission date of a paper, e.g., when the authors
upload the paper much later to arXiv. Thus, we conduct a manual evaluation to
verify if a paper genuinely appeared the first time as indicated by its arXiv
release time stamp. If the paper was available earlier, we remove it from
consideration.

Steps 1 and 2+3 above result in two distinct datasets that we release with
this report. We refer to them as arxiv-0623 and arxiv-0623-top40,
respectively. Table 1 gives elementary statistics on each of them.

No. |  Title |  Cat. |  Link |  Week |  Cit |  z-score
---|---|---|---|---|---|---
1 |  LLaMA: Open and Efficient Foundation Language Models |  cs.CL |  http://arxiv.org/abs/2302.13971v1 |  9 |  874 |  28.051
2 |  GPT-4 Technical Report |  cs.CL |  http://arxiv.org/abs/2303.08774v3 |  11 |  509 |  25.382
3 |  PaLM 2 Technical Report |  cs.CL |  http://arxiv.org/abs/2305.10403v1 |  20 |  82 |  25.182
4 |  Sparks of Artificial General Intelligence: Early experiments with GPT-4 |  cs.CL |  http://arxiv.org/abs/2303.12712v5 |  12 |  354 |  24.302
5 |  PaLM-E: An Embodied Multimodal Language Model |  cs.LG |  http://arxiv.org/abs/2303.03378v1 |  10 |  164 |  21.225
6 |  QLoRA: Efficient Finetuning of Quantized LLMs |  cs.LG |  http://arxiv.org/abs/2305.14314v1 |  21 |  30 |  19.944
7 |  Segment Anything |  cs.CV |  http://arxiv.org/abs/2304.02643v1 |  14 |  165 |  18.548
8 |  Judging LLM-as-a-judge with MT-Bench and Chatbot Arena |  cs.CL |  http://arxiv.org/abs/2306.05685v2 |  23 |  21 |  17.916
9 |  A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity |  cs.CL |  http://arxiv.org/abs/2302.04023v2 |  6 |  214 |  16.819
10 |  A Survey of Large Language Models |  cs.CL |  http://arxiv.org/abs/2303.18223v11 |  13 |  169 |  16.594
11 |  Visual Instruction Tuning |  cs.CV |  http://arxiv.org/abs/2304.08485v1 |  16 |  89 |  15.277
12 |  Tree of Thoughts: Deliberate Problem Solving with Large Language Models |  cs.CL |  http://arxiv.org/abs/2305.10601v1 |  20 |  49 |  14.968
13 |  Voyager: An Open-Ended Embodied Agent with Large Language Models |  cs.AI |  http://arxiv.org/abs/2305.16291v1 |  21 |  21 |  13.860
14 |  Toolformer: Language Models Can Teach Themselves to Use Tools |  cs.CL |  http://arxiv.org/abs/2302.04761v1 |  6 |  175 |  13.716
15 |  How Close is ChatGPT to Human Experts? Comparison Corpus, Evaluation, and Detection |  cs.CL |  http://arxiv.org/abs/2301.07597v1 |  3 |  94 |  13.712
16 |  Extracting Training Data from Diffusion Models |  cs.CR |  http://arxiv.org/abs/2301.13188v1 |  5 |  97 |  13.596
17 |  Large Language Models are not Fair Evaluators |  cs.CL |  http://arxiv.org/abs/2305.17926v1 |  22 |  14 |  13.352
18 |  HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face |  cs.CL |  http://arxiv.org/abs/2303.17580v3 |  13 |  129 |  12.614
19 |  A Watermark for Large Language Models |  cs.LG |  http://arxiv.org/abs/2301.10226v3 |  4 |  76 |  12.481
20 |  DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvature |  cs.CL |  http://arxiv.org/abs/2301.11305v2 |  4 |  76 |  12.481
Table 2: Papers, their prime category, arXiv link, week of first arXiv submission, citation count (as of 07/29/2023) and z-score. Top 20 papers according to z-score among all arxiv-0623 papers. No. |  Title |  Cat. |  Link |  Week |  Cit |  z-score
---|---|---|---|---|---|---
21 |  Mastering Diverse Domains through World Models |  cs.AI |  http://arxiv.org/abs/2301.04104v1 |  2 |  59 |  12.238
22 |  Augmented Language Models: a Survey |  cs.CL |  http://arxiv.org/abs/2302.07842v1 |  7 |  79 |  12.079
23 |  A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT |  cs.AI |  http://arxiv.org/abs/2302.09419v3 |  7 |  79 |  12.079
24 |  ImageBind: One Embedding Space To Bind Them All |  cs.CV |  http://arxiv.org/abs/2305.05665v2 |  19 |  39 |  11.966
25 |  Muse: Text-To-Image Generation via Masked Generative Transformers |  cs.CV |  https://arxiv.org/abs/2301.00704 |  1 |  111 |  11.692
26 |  T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Models |  cs.CV |  http://arxiv.org/abs/2302.08453v2 |  7 |  76 |  11.609
27 |  Is ChatGPT a General-Purpose Natural Language Processing Task Solver? |  cs.CL |  http://arxiv.org/abs/2302.06476v2 |  6 |  145 |  11.328
28 |  SemEval-2023 Task 2: Fine-grained Multilingual Named Entity Recognition (MultiCoNER 2) |  cs.CL |  http://arxiv.org/abs/2305.06586v2 |  19 |  36 |  11.024
29 |  Mathematical Capabilities of ChatGPT |  cs.LG |  http://arxiv.org/abs/2301.13867v2 |  5 |  79 |  11.016
30 |  The Flan Collection: Designing Data and Methods for Effective Instruction Tuning |  cs.AI |  http://arxiv.org/abs/2301.13688v2 |  5 |  78 |  10.873
31 |  The False Promise of Imitating Proprietary LLMs |  cs.CL |  http://arxiv.org/abs/2305.15717v1 |  21 |  16 |  10.480
32 |  The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only |  cs.CL |  http://arxiv.org/abs/2306.01116v1 |  22 |  11 |  10.421
33 |  Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes |  cs.CL |  http://arxiv.org/abs/2305.02301v2 |  18 |  26 |  10.387
34 |  Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding |  cs.CL |  http://arxiv.org/abs/2306.02858v3 |  23 |  12 |  10.136
35 |  InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning |  cs.CV |  http://arxiv.org/abs/2305.06500v2 |  19 |  33 |  10.083
36 |  PandaGPT: One Model To Instruction-Follow Them All |  cs.CL |  http://arxiv.org/abs/2305.16355v1 |  21 |  15 |  9.804
37 |  ChatGPT is not all you need. A State of the Art Review of large Generative AI models |  cs.LG |  http://arxiv.org/abs/2301.04655v1 |  2 |  46 |  9.459
38 |  Theory of Mind May Have Spontaneously Emerged in Large Language Models |  cs.CL |  http://arxiv.org/abs/2302.02083v3 |  5 |  68 |  9.440
39 |  mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality |  cs.CL |  http://arxiv.org/abs/2304.14178v1 |  17 |  34 |  9.377
40 |  Otter: A Multi-Modal Model with In-Context Instruction Tuning |  cs.CV |  http://arxiv.org/abs/2305.03726v1 |  18 |  23 |  9.146
Table 3: Papers, their prime category, arXiv link, week of first arXiv
submission, citation count (as of 07/29/2023) and z-score. Papers 21-40
according to z-score among all arxiv-0623 papers.

##  3 Top $N$ papers

Table 2 showcases the top 20 papers extracted according to the methodology
described in Section 2. We make several interesting observations:

  * •

13 out of 20 (65%) of papers have cs.CL as their prime arXiv category (note
that authors of papers may wish to indicate as many additional categories as
they desire). cs.LG is the prime category 3 times, followed by cs.CV (computer
vision; 2 times) and cs.CR (cryptography) and cs.AI (1 time each).

  * •

The absolute citation counts vary drastically, with 14 as lowest number in our
top-20 list for a paper published in very late May (_Large Language Models are
not Fair Evaluators_ [30]) and 874 as highest numbers for the LLaMA paper [28]
published in late February. The relative citation counts vary from 12 standard
deviations above the mean to 28 standard deviations above the mean.

  * •

The four dominating papers can be seen as technical reports on LLM foundations
models, including LLaMA [28] (the paper with the highest z-score), PaLM 2 [2],
and GPT4 (represented twice; once as an OpenAI publication without dedicated
authors focusing on technical details [24] and once by a group of Microsoft
researchers focused on extensive evaluation [6], both published at around the
same time). A “Survey of Large Language Models” [32] (rank 10 in our list)
published in late March and already updated 11 times further indicates the
popularity of diverse LLMs.

  * •

While not all being technical reports or surveys, the vast majority of top
papers are centered around LLMs (at least 18 out of 20, i.e., 90%). Exceptions
are two papers from the computer vision domain (ranks 7 and 13).

  * •

It is interesting that LLaMA [28], a set of efficient (and open-source)
foundation language models, dominates overall. This hints at the importance of
efficiency for LLMs in general, both from an environmental perspective but
possibly even more so from a practical perspective, as the LLaMA models can
still be fine-tuned even by researchers with a ‘modest’ GPU endowment [19].
Efficiency is further represented by QLoRA [8], submitted to arXiv in late
May, which discusses efficient fine-tuning of quantized LLMs.

  * •

Three top papers [3, 16, 27] (ranks 9, 15 and 18) are specifically centered
around ChatGPT (arguably as the originator of the new LLM hype [20]) and
particularly discuss its _evaluation_ including failure cases. The paper [27]
uses ChatGPT to solve AI tasks by querying huggingface.

  * •

Two further top papers (ranks 12 and 14) explore problem solving with LLMs,
one using external tools [25] and one using reasoning strategies [31].

  * •

Using LLMs for evaluation is discussed in the two papers [30, 33] (ranks 8 and
17), one for evaluating open-ended dialogue and one discussing biases of
evaluation with LLMs. Both papers are much more recent, being published in
late May and early June.

  * •

Two papers [9, 29] (ranks 5 and 13) discuss embodied agents that can interact
with the real world, making use of LLMs.

  * •

Two papers [17, 22] (ranks 19 and 20) can be seen as particularly discussing
the ethical aspects of detecting LLM generated text (e.g., for spotting
misleading AI generated content or to detect cheating in educational contexts)
and watermarking AI generated text, i.e., embedding signals in automatically
generated text that allow its algorithmic detection. Both papers were
published early on, in late January.

  * •

Finally, the exceptions in our top 20 list are two computer vision papers. The
_Segment Anything_ paper [18] by Meta AI Research provides a dataset for image
segmentation. The paper [7] discusses privacy of image diffusion models such
as DALL-E 2 (which can be considered the analogues of LLMs in the computer
vision domain). A further computer vision paper introduces a multimodal
framework called LLaVA [21], building on top of GPT4.

  * •

Recently, there has been a debate whether AI/NLP has become more negative,
i.e., whether papers tend to report more negatively regarding ongoing research
(e.g., outline limitations and failure cases) [5, 4]. In our top-20 list, only
two papers (10%) could be considered critique papers, namely [30], which
focuses on and uncovers biases in LLMs as evaluation models, and [7], which
criticizes lack of privacy of diffusion models, allowing to retrieve private
information from the training data. In the top-40 list, there are two
additional negative papers, i.e., [12] which disputes the mathematical
capabilities of ChatGPT, and [15], which challenges whether distillation in
which a smaller student LLM is trained on the outputs of a larger properietary
LLM such as ChatGPT is really effective. A few papers are partly negative,
highlighting some limitations, such as [3]. Overall, the most popular papers
are (currently) thus positive regarding the development and abilities of
recent LLMs.

Table 3 gives analogous papers with rank 21 to 40. We refrain from an in-depth
analysis as above. The papers have a similar scope, however, with 11 out of 20
(55%) having cs.CL as primary category and 13 out of 20 (65%) having a variant
of LLM in their title (language models, ChatGPT, GPT, etc.). Interestingly,
the list of papers with ranks 21-40 contain quite a few multimodal approaches
such as text-to-image generation models, and relatively more so than the list
of papers with ranks 1-20.

##  4 Analysis

We now briefly perform a few further analyses on our corpus (not only
arxiv-0623-top40 but also arxiv-0623) in order to better understand recent
developments.

### How many citations and standard deviations are there per week?

Figure 1 gives the mean citation counts of papers belonging to three primary
categories (cs.CL, cs.LG, and all others) over time. We observe that:

  * •

citations tend to decrease over time (as is expected; more recent papers
cannot yet have been cited so frequently), with, on average, decisively fewer
than 2 citations per paper starting from May for all three arXiv categories

  * •

cs.CL attracts (considerably) more citations than cs.LG and the aggregation of
all other involved primary categories

  * •

February has been the month with the most impactful papers in cs.CL,
especially week 6 (e.g., Toolformer [25] and ChatGPT analysis [3] submitted to
arXiv) and week 9 (e.g., LLaMA [28] submitted)

Detailed results including overall standard deviations are also give in Table
4. Standard deviations are particularly large in weeks 1, 6, 8-13.

Figure 1: Mean number of citations over weeks for different arXiv categories. Week Number | Week Date | Mean | Std | Mean cs.CL | Mean cs.LG | Mean Rest
---|---|---|---|---|---|---
1 | 01-01/01-07 | 2.208 | 9.305 | 3.164 | 1.175 | 2.787
2 | 01-08/01-14 | 1.759 | 4.677 | 1.714 | 1.852 | 1.703
3 | 01-15/01-21 | 2.029 | 6.707 | 4.110 | 1.707 | 1.452
4 | 01-22/01-28 | 1.942 | 5.934 | 2.318 | 1.729 | 2.084
5 | 01-29/02-04 | 2.144 | 6.977 | 4.314 | 1.417 | 2.290
6 | 02-05/02-11 | 2.647 | 12.566 | 7.578 | 1.464 | 2.036
7 | 02-12/02-18 | 1.900 | 6.383 | 3.368 | 1.181 | 2.112
8 | 02-19/02-25 | 2.158 | 9.813 | 3.617 | 1.321 | 2.618
9 | 02-26/03-04 | 2.651 | 31.063 | 10.496 | 0.983 | 1.426
10 | 03-05/03-11 | 1.688 | 7.647 | 3.734 | 1.442 | 1.224
11 | 03-12/03-18 | 2.191 | 19.967 | 7.901 | 0.947 | 1.196
12 | 03-19/03-25 | 1.930 | 14.487 | 6.117 | 0.764 | 1.286
13 | 03-26/04-01 | 2.197 | 10.052 | 5.688 | 1.104 | 1.485
14 | 04-02/04-08 | 1.743 | 8.802 | 3.380 | 0.744 | 1.761
15 | 04-09/04-15 | 1.680 | 6.175 | 4.000 | 0.585 | 1.692
16 | 04-16/04-22 | 1.692 | 5.715 | 3.093 | 0.957 | 1.471
17 | 04-23/04-29 | 1.174 | 3.501 | 2.368 | 0.704 | 1.020
18 | 04-30/05-06 | 0.885 | 2.418 | 1.320 | 0.633 | 0.718
19 | 05-07/05-13 | 0.869 | 3.187 | 1.162 | 0.378 | 1.024
20 | 05-14/05-20 | 0.641 | 3.231 | 1.204 | 0.270 | 0.431
21 | 05-21/05-27 | 0.496 | 1.479 | 0.550 | 0.485 | 0.405
22 | 05-28/06-03 | 0.334 | 1.023 | 0.494 | 0.251 | 0.306
23 | 06-04/06-10 | 0.276 | 1.157 | 0.474 | 0.162 | 0.260
24 | 06-11/06-17 | 0.338 | 1.674 | 0.449 | 0.405 | 0.192
25 | 06-18/06-24 | 0.309 | 3.861 | 0.331 | 0.481 | 0.123
26 | 06-25/07-01 | 0.257 | 2.163 | 0.228 | 0.173 | 0.354
Table 4: Mean number of citations, over all papers including standard
deviations, and for the primary categories cs.CL, cs.LG and the remaining
categories.

### How many arXiv categories (scientific subfields) are involved?

Our dataset arxiv-0623 comprises 20,843 papers submitted to arXiv between
01/01/2023 and 06/31/2023 with at least one of the indicated categories given
as cs.CL or cs.LG. As NLP and ML affect all aspects of life nowadays, we would
expect that these papers do not only originate from either ML or NLP. Indeed,
we find that our 20,843 papers are assigned to 123 different primary arXiv
categories. We give detailed statistics on those 19 primary categories
occurring at least 100 times in Table 5. Overall, the most frequent 19 primary
categories are made up of 5 top level categories, namely: cs (computer
science), stat (statistics), eess (electrical engineering and systems
science), math (mathematics) and quant-ph (quantum physics).555ArXiv does
unfortunately not include the humanities or social sciences directly. The five
most frequent fine-grained categories are cs.LG, cs.CL, cs.CV (computer
vision), stat.ML (statistics, machine learning) and cs.AI (artificial
intelligence).

A pie chart of the distribution of primary categories is shown in Figure 2.
cs.LG is the largest category, almost 40% of papers have it as its primary
category. cs.CL is only about half the size (but dominates the top-40 papers
as discussed above). Other primary categories (outside of the top 5
categories) are about the same size as cs.CL.

Category | Occurrences
---|---
cs.LG | 8127
cs.CL | 4966
cs.CV | 1670
stat.ML | 859
cs.AI | 455
eess.IV | 414
cs.CR | 304
cs.IR | 288
cs.RO | 285
cs.SD | 265
math.OC | 214
eess.AS | 212
eess.SP | 201
cs.HC | 148
cs.NE | 143
eess.SY | 134
cs.SE | 127
quant-ph | 125
cs.CY | 111
Table 5: All primary categories given in our arXiv dataset whose occurrence
exceeds 100. ArXiv categories are described here:
https://arxiv.org/category_taxonomy. Figure 2: Pie chart of distribution of
main categories in our dataset.

### What distinguishes top papers from other papers?

We use the tool of [13] based on the log-likelihood ratio test [10] to
determine unusually frequent words in our top-40 papers arxiv-0623-top40 vs.
all other papers. Among the top-10 most distinctive unigrams are _chatgpt,
gpt-4, modalities, visual, zero-shot_. Among the top bigrams are _language
models, large language, models (llms), wide range_. The singular most
important trigram is _large language models_. Conversely, words that
characterize papers outside the top-40 the best are jargon referring to an
older deep learning era such as _learning, neural, deep, network, neural
network, machine learning_ , etc. While this characterization is very
simplistic (it certainly does not satisfy to publish a paper on LLMs to obtain
high citation numbers), it is nonetheless insightful.

Top-40 papers also have way more authors on average (11.8, with a standard
deviation of 19.5) compared to the remaining papers (4.5 with a standard
deviation of 3.2). Part of the effect could be trivial: more authors can
increase self-citation counts (an arguably at least partly unethical practice
[26]). On the other hand, more fundamental research may require a larger
author list and industry may also produce papers with a higher number of
authors.

Figure 3: Wordcloud based on the top-40 papers.

### What are the most important key words of the top-40 papers?

We plot a wordcloud of the top-40 papers (see Figure 3). To do so, we use
KeyBERT [14] to identify the 5 most important tri-grams from the title and
abstract of each paper. Then we filter out a manually selected list of
unimportant words and lemmatize each word. Finally, we use the python library
wordcloud666https://github.com/amueller/word_cloud for plotting. Here the
focus of current research into ever larger models becomes apparent again, with
phrases such as trillion token, 175b, large scale and large language model.
The keywords publicly available also show a focus on non-proprietary data and
models.

### How popular are LLMs over time in our arXiv dataset?

While we have seen that LLMs are the dominating theme in the top-40 paper
list, we wonder how the popularity of LLMs and ChatGPT have developed over
time in our complete arXiv dataset arxiv-0623. To this end, we query the
keywords “LLMs” and “ChatGPT” in our dataset over time and flag a paper as
relevant if it contains the keywords in its title or abstract.777We lowercase
abstracts and titles, and we look for the keywords “llm(s)” and “large
language model(s)” for LLM; for ChatGPT, we look for “chatgpt” and “chat-gpt”.

Figure 4 shows the results. Both keywords were not very relevant in early
2023, less than 2% of papers contained them in January. The ChatGPT curve
increases until late March (6% of all papers). Starting from mid-April, LLMs
become the more popular keyword. ChatGPT as a keyword declines since then,
while LLMs spike in the week of 05/21 (which marks 2023’s camera-ready
submission deadline for the popular NLP conference ACL
https://www.aclweb.org/portal/content/acl-2023-call-papers) with almost 12% of
papers containing it; we assume that many accepted ACL papers (with LLMs as a
topic) were posted to arXiv right after the camera-ready deadline. Since then,
LLMs seem to be declining as a keyword, also — even though this could just be
an artefact of the conference deadline.

Figure 4: Popularity of ChatGPT and LLMs (in percentage of papers having the
words in their abstracts or titles) over time in our dataset.

##  5 Conclusion

We have examined arXiv papers related to the categories cs.CL and cs.LG over
the first half of 2023. First, we sorted papers according to their normalized
citation counts, finding that LLM related papers clearly dominate. Within
LLMs, the most popular current issues center around: efficiency, LLM based
evaluation, ethical aspects, embodied agents and problem solving with LLMs
(only slightly less prominent are multimodal approaches encompassing language
and other modalities such as images, with at least 8 papers within the
top-40). We have also looked at, among others: (i) what characteristics top
papers have relative to papers outside the top-40 list in terms of number of
authors and vocabulary, (ii) the distributions of citations in our dataset,
and (iii) the popularity of ChatGPT, which ‘caused’ the current hype
surrounding LLMs in late 2022, and LLMs over time. We hope that our
investigation is beneficial not only to newcomers and outsiders to the field
of NLP and ML (of which there are seemingly very many nowadays, given how
popular the fields have become [34]), providing quick links to useful starting
literature, but also to established researchers and their doctoral students.

In the future, we want to regularly update the current report to see how
tastes shift over time, examine our arXiv datasets arxiv-0623 and
arxiv-0623-top40 in much more depth, and include further arXiv categories
related to AI fields (e.g., cs.CV, stat.ML, cs.AI) into our datasets, among
others.

## Limitations

Limitations of our approach include the following. First of all, science tools
like SemanticScholar or GoogleScholar make quite a few mistakes in correctly
attributing citations. While we did not study this in depth, we note for
example that LLaMA (our top paper) has 874 citations according to
SemanticScholar (July 29, 2023) but only 710 citations according to
GoogleScholar, a relative difference of $\frac{164}{874}=18.7\%$. The paper
with fewest citations in our top 20 list [30] has 14 citations (July 29, 2023)
according to SemanticScholar but only 9 citations according to GoogleScholar,
a relative difference of $\frac{5}{14}=35.7\%$. While we do think that our
rankings are relatively reliable, such deviations may naturally bias our
selection of papers, assumedly with higher uncertainty for low citation
papers. Secondly, focusing particularly on highly cited papers may induce a
bias towards these papers similar to that of a self-fulfilling prophecy or
preferential attachment. Thirdly, our focus on weekly citation averages may
have unexpected effects: for example, a younger paper with more citations
could be ranked below an older paper with fewer citations, for example, if
that older paper was published in a week with fewer average citations (e.g.,
in the early weeks of January where research, and other human activity, is
typically less productive, at least in relevant parts of the world, due to
preceding holiday activities). Finally, some authors and research groups,
potentially more traditional ones, may refrain from submitting their papers to
arXiv, despite its otherwise high popularity particularly in the computer
science community (see exponential submission growth rates of arXiv submission
numbers in the last decades
https://info.arxiv.org/help/stats/2021_by_area/index.html). Papers from such
authors or groups will not be part of our dataset and analysis.

Our limitations must be kept in mind when interpreting our results.

## Acknowledgements

The NLLG group gratefully acknowledges support from the Federal Ministry of
Education and Research (BMBF) via the interdisciplinary AI research grant
“Metrics4NLG”. Steffen Eger is further supported by the DFG Heisenberg grant
EG 375/5-1. We thank Andreas “Max Power” Rücklé for thoughtful discussions.

## References

  * [1] Dag W. Aksnes, Liv Langfeldt, and Paul Wouters.  Citations, citation indicators, and research quality: An overview of basic concepts and theories.  SAGE Open, 9(1):2158244019829575, 2019.
  * [2] Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent El Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Yi Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag, Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Saeta, Rajkumar Samuel, Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter, Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu.  Palm 2 technical report, 2023.
  * [3] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung.  A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity, 2023.
  * [4] Dominik Beese, Begüm Altunbaş, Görkem Güzeler, and Steffen Eger.  Did AI get more negative recently?  Royal Society Open Science, 10(3):221159, 2023.
  * [5] Samuel Bowman.  The dangers of underclaiming: Reasons for caution when reporting how NLP systems fail.  In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7484–7499, Dublin, Ireland, May 2022. Association for Computational Linguistics.
  * [6] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang.  Sparks of artificial general intelligence: Early experiments with gpt-4, 2023.
  * [7] Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace.  Extracting training data from diffusion models, 2023.
  * [8] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer.  Qlora: Efficient finetuning of quantized llms, 2023.
  * [9] Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence.  Palm-e: An embodied multimodal language model, 2023.
  * [10] Ted Dunning.  Accurate methods for the statistics of surprise and coincidence.  Computational Linguistics, 19(1):61–74, 1993.
  * [11] Steffen Eger, Chao Li, Florian Netzer, and Iryna Gurevych.  Predicting research trends from arxiv.  arXiv preprint arXiv:1903.02831, 2019.
  * [12] Simon Frieder, Luca Pinchetti, Alexis Chevalier, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Christian Petersen, and Julius Berner.  Mathematical capabilities of chatgpt, 2023.
  * [13] Yang Gao, Steffen Eger, Ilia Kuznetsov, Iryna Gurevych, and Yusuke Miyao.  Does my rebuttal matter? insights from a major NLP conference.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1274–1290, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [14] Maarten Grootendorst.  Keybert: Minimal keyword extraction with bert., 2020.
  * [15] Arnav Gudibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song.  The false promise of imitating proprietary llms, 2023.
  * [16] Biyang Guo, Xin Zhang, Ziyuan Wang, Minqi Jiang, Jinran Nie, Yuxuan Ding, Jianwei Yue, and Yupeng Wu.  How close is chatgpt to human experts? comparison corpus, evaluation, and detection, 2023.
  * [17] John Kirchenbauer, Jonas Geiping, Yuxin Wen, Jonathan Katz, Ian Miers, and Tom Goldstein.  A watermark for large language models, 2023.
  * [18] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick.  Segment anything, 2023.
  * [19] Ji-Ung Lee, Haritz Puerto, Betty van Aken, Yuki Arase, Jessica Zosa Forde, Leon Derczynski, Andreas Rücklé, Iryna Gurevych, Roy Schwartz, Emma Strubell, and Jesse Dodge.  Surveying (dis)parities and concerns of compute hungry nlp research, 2023\.
  * [20] Christoph Leiter, Ran Zhang, Yanran Chen, Jonas Belouadi, Daniil Larionov, Vivian Fresen, and Steffen Eger.  Chatgpt: A meta-analysis after 2.5 months.  ArXiv, abs/2302.13795, 2023.
  * [21] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.  Visual instruction tuning, 2023.
  * [22] Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn.  Detectgpt: Zero-shot machine-generated text detection using probability curvature, 2023.
  * [23] Mark EJ Newman.  Prediction of highly cited papers.  Europhysics Letters, 105(2):28002, 2014.
  * [24] OpenAI.  Gpt-4 technical report, 2023.
  * [25] Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.  Toolformer: Language models can teach themselves to use tools, 2023.
  * [26] Marco Seeber, Mattia Cattaneo, Michele Meoli, and Paolo Malighetti.  Self-citations as strategic response to the use of metrics for career decisions.  Research Policy, 48(2):478–491, 2019.
  * [27] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang.  Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face, 2023.
  * [28] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample.  Llama: Open and efficient foundation language models, 2023.
  * [29] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar.  Voyager: An open-ended embodied agent with large language models, 2023\.
  * [30] Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui.  Large language models are not fair evaluators, 2023.
  * [31] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan.  Tree of thoughts: Deliberate problem solving with large language models, 2023.
  * [32] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen.  A survey of large language models, 2023.
  * [33] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.  Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
  * [34] Caleb Ziems, William B. Held, Omar Shaikh, Jiaao Chen, Zhehao Zhang, and Diyi Yang.  Can large language models transform computational social science?  ArXiv, abs/2305.03514, 2023.


classes. Thus, we have isolated the $G_{2b^{\prime\prime};\gamma,\lambda,c}$
isometry class.

  13. 13.

We are in the formfactor class
$f(\zeta,u)=A(u)\ln\zeta+A_{1}(u)\zeta+A_{0}(u)$. The condition

$\frac{1}{I_{\mathrm{d}}^{(i\kappa)}}=\alpha
e^{i\gamma}\quad\iff\quad\frac{A^{3}}{\dot{A}^{2}}=\alpha e^{i\gamma},$ (142)

with external parameters $\alpha>0$, $\gamma\in[0,\pi]$, integrates to
$A(u)=\frac{4\alpha}{(u-u_{0})^{2}}e^{i\gamma}$, where we can set $u_{0}=0$ by
gauge freedom. Note that the any value of $\kappa$ produces the same result
and $I_{\mathrm{d}}^{(i\kappa)}$ is well-defined in (36) because we are in the
$I_{\mathrm{c}}=\frac{1}{2}$ branch.

Then, taking into account the evaluation in (49), the condition

$\mathbf{M}^{(i\kappa)}=0\quad\iff\quad-\frac{4}{u\bar{\zeta}}(u\dot{A}_{1}-(i\kappa-2)A_{1})\bm{\ell}\bm{\ell}\bm{\ell}=0,$
(143)

with external parameter $\kappa\in\mathbb{R}$ that now enters non-trivially,
integrates to $A_{1}(u)=A_{1,0}u^{i\kappa-2}$. We can set $A_{1,0}=c>0$ by the
$\zeta\mapsto e^{i\beta}\zeta$ gauge freedom, while the $f\mapsto\bar{f}$
freedom has already been fixed by the requirement that $\gamma\in[0,\pi]$. If
either of the above two conditions fails, we end up in the generic $G_{1}$
class. Otherwise, relying on the evaluation in (47) and using the following
formula to extract the invariant parameter

$c=|A_{1,0}|=\frac{|N|}{|\mathbf{K}|},$ (144)

we have isolated the $G_{2a^{\prime};\alpha,\gamma,\kappa,c}$ isometry class.

  14. 14.

By the evaluation (29), the condition $\mathfrak{L}=0$ is equivalent to

$\left(\frac{(\ln f_{,\zeta\zeta})_{,u}}{(\ln
f_{,\zeta\zeta})_{,\zeta}}\right)_{\zeta}=0.$ (145)

This equation has already been integrated in IC.9 and IC.17 around (79) and
(113). Since the $I_{\mathrm{c}}=-(1/(\ln
f_{,\zeta\zeta})_{,\zeta})=\frac{1}{2}$ possibility has already been handled
in another branch, we can assume that $I_{\mathrm{c}}\neq\frac{1}{2}$ and as a
result $f(\zeta,u)=\mathrm{f}(\zeta)+A_{1}(u)\zeta+A_{0}(u)$. The additional
condition $\mathbf{M}^{(\infty)}=0$, relying on evaluation (51), is equivalent
to

$\mathrm{f}^{\prime\prime\prime}(z)=0\quad\text{or}\quad\dot{A}_{1}=0.$ (146)

The first possibility has already been handled in a previous branch where
$\overline{\mathbf{K}}\cdot\mathbf{K}=0$, so we can assume
$\mathrm{f}^{\prime\prime\prime}(z)\neq 0$, which then restricts
$A_{1}(u)=A_{1,0}$ constant. Recalling that $A_{0}(u)$ is pure gauge, we end
up in the formfactor class

$f(\zeta,u)=\mathrm{f}(\zeta).$ (147)

We do not need to separately consider the form
$\mathrm{f}(\zeta)+A_{1,0}\zeta$ because we can simply redefine
$\mathrm{f}(z)$ to absorb the term proportional to $A_{1,0}$.

If $\mathfrak{L}=0$, but $\mathbf{M}^{(\infty)}\neq 0$, while
$I_{\mathrm{c}}\neq\frac{1}{2}$, then the underlying geometry is not one of
the highly symmetric pp-wave classes, which lands us in the generic $G_{1}$
class.

It remains to examine the possibility that $\mathfrak{L}\neq 0$.

  15. 15.

Within the formfactor class $f(\zeta,u)=\mathrm{f}(\zeta)$, relying on the
evaluation (50) with $\lambda=0$, consider the condition

$I_{\mathrm{e}}^{(\infty)}=\frac{\mathrm{f}^{\prime\prime\prime}(\zeta)\mathrm{f}^{\prime}(\zeta)}{\mathrm{f}^{\prime\prime}(\zeta)^{2}}=\frac{k-2}{k-1},$
(148)

where the expression on the right-hand side is just a convenient way for us to
parametrize a complex constant. To exhaust all the possible values of the
constant, we must include $k=\infty$ ($I_{\mathrm{e}}^{(\infty)}=1$) and
obviously exclude $k=1$.

When $k=\infty$, the corresponding equation has been integrated in IC.18,
which isolates the $G_{3c;\lambda}$ isometry class, with
$\lambda=\frac{1}{2}|\mathbf{K}|>0$, using the evaluation (22).

When $k\neq\infty$, the integration in IC.19 yields two special values $k=0$
and $k=2$ (with $k=1$ already excluded above), and the special subfamily
$k=2i\kappa$, $\kappa\in\mathbb{R}\setminus\\{0\\}$. The possibility $k=0$ is
already handled in the $I_{\mathrm{c}}=\frac{1}{2}$ branch, while the
possibility $k=2$ is already handled in the
$\overline{\mathbf{K}}\cdot\mathbf{K}=0$ branch. As discussed in IC.20, the
special subfamily $k=2i\kappa$ isolates the $G_{3d;\gamma,\kappa}$ isometry
class, with $\gamma\in(0,\pi)$ uniquely fixed by

$e^{2i\gamma}=\frac{(2i\kappa+1)(i\kappa-1)^{2}}{(2i\kappa-1)(i\kappa+1)^{2}}\left|\frac{\mathbf{K}}{2(i\kappa-1)}\right|^{4i\kappa}J,$
(149)

relying on the evaluation in (54). This invariant isolates $\gamma$ in every
case except when $\gamma=0$ or $\pi$. In that case both values of $\gamma$ are
fixed by complex conjugation and give $e^{2i\gamma}=1$. We have found that it
is not possible to isolate $e^{i\gamma}$ directly using tensor invariants.
However, using the Cartan-Karlhede algorithm, it is possible to isolate
$e^{i\gamma}=\pm 1$ and the corresponding isometry classes are definitely
distinct. So for this particular formfactor and choice of $\gamma$, the IDEAL
characterization is unable to distinguish between them.

The remaining values of $k\neq i\mathbb{R}\cup\\{1,2\\}$, according to the
discussion in IC.19 isolates the $G_{2b^{\prime\prime\prime};k}$ isometry
class.

It remains to explore the possibility that $I_{\mathrm{e}}^{(\infty)}\neq$
const.

  16. 16.

Within the $f(\zeta,u)=\mathrm{f}(\zeta)$ formfactor family,
$I_{\mathrm{e}}^{(\infty)}$ depends holomorphically on $\zeta$, as we see from
(148). Further, in this branch, we can assume that
$I_{\mathrm{e}}^{(\infty)}\neq$ const. Then, from evaluation (22),
$|\mathbf{K}|^{2}=\overline{\mathcal{K}}\mathcal{K}$, where
$\mathcal{K}=\mathrm{f}^{\prime\prime\prime}(\zeta)/\mathrm{f}^{\prime\prime}(\zeta)$
obviously also depends holomorphically on $\zeta$. By the holomorphic implicit
function theorem, there must then exist some holomorphic function $F(y)$ such
that $\ln\mathcal{K}=F(I_{\mathrm{e}}^{(\infty)})$, which coincides with the
signature relation (116) in IC.17. Expressed in terms of the $|\mathbf{K}|$
invariant the relation becomes

$\ln(|\mathbf{K}|^{2})=2\operatorname{Re}F\left(I_{\mathrm{e}}^{(\infty)}\right),$
(150)

where we recognize the harmonic function $\operatorname{Re}F(y)$ as one of the
invariant isometry class parameters discussed in IC.17. The remaining
invariant parameter discussed in IC.17 is isolated by the following relation,
where the function $\mathcal{G}_{\operatorname{Re}F}(w)$ from (122) is
determined by $\operatorname{Re}F$ (through holomorphic extension and solving
an ODE),

$\mathcal{G}_{\operatorname{Re}F}\left(I_{\mathrm{e}}^{(\infty)}\right)=\cos\gamma\in[-1,1]$
(151)

with the right-hand side constant. Together, these parameters isolate the
isometry class $G_{2b^{\prime};\gamma,\operatorname{Re}F}$, with
$\gamma\in[0,\pi]$ uniquely fixed by the value $\cos\gamma\in[-1,1]$.

  17. 17.

In this branch, we only have the inequalities
$\overline{\mathbf{K}}\cdot\mathbf{K}\neq 0$, $\mathfrak{L}\neq 0$ and
$I_{\mathrm{c}}-\frac{1}{2}\neq 0$s. To define further invariants that we
could use for further analysis, as indicated in Section 3.3, we also need to
eliminate the condition

$k(I_{\mathrm{c}}-\tfrac{1}{2})\neq
I_{\mathrm{c}}\quad\implies\quad\left(\frac{1}{(\ln
f_{,\zeta\zeta})_{,\zeta}}\right)_{,\zeta}=-\frac{k}{2(k-1)}$ (152)

for some complex constant $k$, where we have used the evaluation in (28). The
general 4-parameter solution for different values of $k$ (interpreting the
special value $k=\infty$ as $I_{\mathrm{c}}-\frac{1}{2}=0$, and $k=0$ as
$I_{\mathrm{c}}=0$) is given by

$f(\zeta,u)=\begin{cases}A\ln(\zeta-h)+A_{1}\zeta+A_{0}&(k=\infty),\\\
e^{A(\zeta-h)}+A_{1}\zeta+A_{0}&(k=0),\\\
A(\zeta-h)^{2/k}+A_{1}\zeta+A_{0}&(k\neq 0,\infty),\end{cases}$ (153)

where the integration parameters $A$, $A_{1}$, $A_{0}$ and $h$ are of course
$u$-dependent. None of these forms belong to our catalog of highly symmetric
pp-wave geometries in (44), unless they have already been handled in other
branches of our decision diagram in Figure 2, namely node 6 for the logaritmic
form or node 15 for the exponential and power forms.

Thus, if the equality in (152) is actually satisfied, the pp-wave geometry
must belong belong to the generic $G_{1}$ class, and it remains to explore the
possibility where the equality does not hold.

  18. 18.

Within this branch, the $\mathbf{L}^{(k)}$ invariant is well-defined. Recall
the evaluation $\mathbf{L}^{(k)}=-2\Xi^{(k)}\bm{\ell}$ in (34), where
$\Xi^{(k)}$ is evaluated in (35). Recall also that in our normalization, the
null vector $\bm{\ell}$ is covariantly constant, which makes it easy to
evaluate the covariant condition

$2\nabla\mathbf{L}^{(k)}=\mathbf{L}^{(k)}\mathbf{L}^{(k)}\quad\iff\quad(\Xi^{(k)})_{,\zeta}=0,\quad(\Xi^{(k)})_{,u}+(\Xi^{(k)})^{2}=0.$
(154)

These differential equations are easily integrated to the general solution

$\Xi^{(k)}=\frac{1}{u-u_{0}},$ (155)

where $u_{0}$ is a real integration constant.

Let us suppose that condition (154) is satisfied with $k=i\kappa$,
$\kappa\in\mathbb{R}$. For completeness, we must also consider the possibility
that $\kappa=\infty$, which means that the limit
$\mathbf{L}^{(\infty)}=\lim_{k\to\infty}\mathbf{L}^{(k)}$ exists and the
appropriately scaled version of the invariant condition (154) is satisfied. We
will examine the $\kappa=\infty$ possibility separately.

Provided $\kappa\neq\infty$, using the evaluation in (35) and clearing the
denominators in $\Xi^{(k)}=1/(u-u_{0})$ straightforwardly results in equation
(96), which was integrated in IC.13. The formfactor $f(\zeta,u)$ then takes
the form

$f(\zeta,u)=\frac{1}{u^{2}}\mathrm{f}(\zeta u^{i\kappa})+A_{1}(u)\zeta,$ (156)

where we have already applied allowed simplification by residual gauge
freedom. A similar integration is possible for other complex values of
$k\in\mathbb{C}\setminus i\mathbb{R}$, but results in a form that can only
belong to the generic $G_{1}$ class. If the condition (154) is not satisfied
at all (including with $\kappa=\infty$), the underlying pp-wave geometry
cannot belong to any of the highly symmetric classes, so it must again belong
to the generic $G_{1}$ class.

Since the invariant $\mathbf{L}^{(i\kappa)}$ is well-defined, so is
$\mathbf{M}^{(i\kappa)}$. According to its evaluation in (49), the following
invariant condition specializes as

$\mathbf{M}^{(i\kappa)}=0\quad\iff\quad u\dot{A}_{1}-(i\kappa-2)A_{1}=0,$
(157)

which results in the form factor $f(\zeta,u)=u^{-2}(\mathrm{f}(\zeta
u^{i\kappa})+A_{1,0}\zeta u^{i\kappa})$, meaning that the $\zeta$-linear term
can just be absorbed by a redefinition of $\mathrm{f}(z)$, which lands us in
the $G_{2a}$ subclass. Otherwise, if condition (157) is not satisfied at all,
the underlying pp-wave geometry cannot belong to any of the highly symmetric
classes, so it must again belong to the generic $G_{1}$ class.

Within the $G_{2a}$ subclass, as evaluated in (48),

$I_{\mathrm{d}}^{(i\kappa)}=-\frac{\mathrm{f}^{\prime\prime\prime}(z)^{2}}{\mathrm{f}^{\prime\prime}(z)^{3}}\quad\text{and}\quad|\mathbf{K}|^{2}=\frac{\overline{\mathrm{f}^{\prime\prime\prime}(z)}}{\overline{\mathrm{f}^{\prime\prime}(z)}}\frac{\mathrm{f}^{\prime\prime\prime}(z)}{\mathrm{f}^{\prime\prime}(z)},$
(158)

where $z=\zeta u^{i\kappa}$. Clearly, $I_{\mathrm{d}}^{(i\kappa)}$ depends
holomorically on $z$, while
$|\mathbf{K}|^{2}=\overline{\mathcal{K}}\mathcal{K}$, where
$\mathcal{K}=\mathrm{f}^{\prime\prime\prime}(z)/\mathrm{f}^{\prime\prime}(z)$
holomorphically depends on $z$.

Consider the possibility that $I_{\mathrm{d}}^{(i\kappa)}=\text{const}$. The
simplest case $I_{\mathrm{d}}^{(i\kappa)}=0$ is equivalent to
$\mathrm{f}^{\prime\prime\prime}(z)=0$ is integrated by the general solution
$\mathrm{f}(z)=Az^{2}+A_{1}z+A_{0}$. All corresponding pp-wave classes fall
into the $G_{5}$ subclass and have already been classified in an earlier
branch (namely node 3) of our decision diagram in Figure 2. Otherwise, the
condition $I_{\mathrm{d}}^{(i\kappa)}\neq 0$ appears as (106), where it has
been integrated in IC.14–16 and shown correspond to either $G_{2a^{\prime}}$
or $G_{3a}$ subclasses, which have also been classified in an earlier branch
(namely nodes 10 and 13) of our decision diagram.

Thus, in this branch we can assume that
$I_{\mathrm{d}}^{(i\kappa)}\neq\text{const}$ and hence by the holomorphic
implicit function theorem, there must then exist some holomorphic function
$F(y)$ such that $\ln\mathcal{K}=F(-I_{\mathrm{d}}^{(i\kappa)})$, which
coincides with the signature relation (99). Expressed in terms of the
$|\mathbf{K}|$ invariant, the relation becomes

$\ln(|\mathbf{K}|^{2})=2\operatorname{Re}F\left(-I_{\mathrm{d}}^{(i\kappa)}\right),$
(159)

where we recognize the harmonic function $\operatorname{Re}F(y)$ as one of the
invariant isometry class parameters discussed in IC.13. The remaining
invariant parameter discussed in IC.13 is isolated by the following relation,
where the function $\mathcal{A}_{\kappa,\operatorname{Re}F}(w)$ from (13) is
determined by $\operatorname{Re}F$ (through holomorphic extension and solving
an ODE),

$\mathcal{A}_{\kappa,\operatorname{Re}F}(I_{\mathrm{d}}^{(i\kappa)},I_{\mathrm{e}}^{(i\kappa)})=c>0$
(160)

with the right-hand side constant. Together, these parameters isolate the
isometry class $G_{2a^{\circ};\kappa,c,\operatorname{Re}F}$.

  19. 19.

Within this branch, the $\mathbf{L}^{(\infty)}$ invariant is well-defined.
Recall the evaluation $\mathbf{L}^{(\infty)}=-2\Xi^{(\infty)}\bm{\ell}$ in
(39), where $\Xi^{(\infty)}$ is evaluated in (40). Recall also that in our
normalization, the null vector $\bm{\ell}$ is covariantly constant, which
makes it easy to evaluate the covariant condition (which is the $k\to\infty$
scaled limit of (154))

$2\nabla\mathbf{L}^{(\infty)}=0\quad\iff\quad(\Xi^{(\infty)})_{,\zeta}=0,\quad(\Xi^{(\infty)})_{,u}=0.$
(161)

These differential equations are easily integrated to the general solution

$\Xi^{(\infty)}=i\lambda,$ (162)

where $\lambda$ is a complex integration constant. Shortly, $\lambda$ will be
restricted to real values.

Let us suppose that condition (161) is satisfied.

Using the evaluation in (40) and clearing the denominators in
$\Xi^{(\infty)}=i\lambda$ straightforwardly results in equation (79), which
was integrated in IC.9. The formfactor $f(\zeta,u)$ then takes the form

$f(\zeta,u)=\mathrm{f}(\zeta e^{i\lambda u})+A_{1}(u)\zeta,$ (163)

where we have already applied allowed simplification by residual gauge
freedom. At this point, we see that only real values of $\lambda$ would be
allowed within the $G_{2b}$ subclass. Hence complex values of
$\lambda\in\mathbb{C}\setminus\mathbb{R}$ result in a form that can only
belong to the generic $G_{1}$ class. Since
$\mathbf{L}^{(\infty)}=-2i\lambda\bm{\ell}$, we can impose
$\lambda\in\mathbb{R}$ by the invariant condition

$\operatorname{Re}\mathbf{L}^{(\infty)}=0.$ (164)

If the condition (161) is not satisfied at all, the underlying pp-wave
geometry cannot belong to any of the highly symmetric classes, so it must
again belong to the generic $G_{1}$ class.

Since the invariant $\mathbf{L}^{(\infty)}$ is well-defined, so is
$\mathbf{M}^{(\infty)}$. According to its evaluation in (51), the following
invariant condition specializes as

$\mathbf{M}^{(\infty)}=0\quad\iff\quad\dot{A}_{1}-i\lambda A_{1}=0,$ (165)

which results in the form factor $f(\zeta,u)=\mathrm{f}(\zeta e^{i\lambda
u})+A_{1,0}\zeta e^{i\lambda u}$, meaning that the $\zeta$-linear term can
just be absorbed by a redefinition of $\mathrm{f}(z)$, which lands us in the
$G_{2b}$ subclass. Otherwise, if condition (165) is not satisfied at all, the
underlying pp-wave geometry cannot belong to any of the highly symmetric
classes, so it must again belong to the generic $G_{1}$ class.

Note that at this point we can assume that $\lambda\neq 0$, since the
$\lambda=0$ was already handled in an earlier branch (namely node 16) in our
decision diagram in Figure 2. Moreover, using the residual gauge
transformation $u\mapsto u/\lambda$, we are free to assume the value
$\lambda=1$.

Within the $G_{2b}$ subclass, as evaluated in (50),

$I_{\mathrm{d}}^{(\infty)}=\lambda^{2}\frac{\mathrm{f}^{\prime\prime\prime}(z)^{2}}{\mathrm{f}^{\prime\prime}(z)^{3}}\quad\text{and}\quad|\mathbf{K}|^{2}=\frac{\overline{\mathrm{f}^{\prime\prime\prime}(z)}}{\overline{\mathrm{f}^{\prime\prime}(z)}}\frac{\mathrm{f}^{\prime\prime\prime}(z)}{\mathrm{f}^{\prime\prime}(z)},$
(166)

where $z=\zeta e^{i\lambda u}$. Clearly, $I_{\mathrm{d}}^{(\infty)}$ depends
holomorically on $z$, while
$|\mathbf{K}|^{2}=\overline{\mathcal{K}}\mathcal{K}$, where
$\mathcal{K}=\mathrm{f}^{\prime\prime\prime}(z)/\mathrm{f}^{\prime\prime}(z)$
holomorphically depends on $z$.

Consider the possibility that $I_{\mathrm{d}}^{(\infty)}=\text{const}$ and
recall that $\lambda\neq 0$. The simplest case $I_{\mathrm{d}}^{(\infty)}=0$
is equivalent to $\mathrm{f}^{\prime\prime\prime}(z)=0$ is integrated by the
general solution $\mathrm{f}(z)=Az^{2}+A_{1}z+A_{0}$. All corresponding pp-
wave classes fall into the $G_{5}$ subclass and have already been classified
in an earlier branch (namely node 3) of our decision diagram in Figure 2.
Otherwise, the condition $I_{\mathrm{d}}^{(\infty)}\neq 0$ appears as (89),
where it has been integrated in IC.10–12 and shown correspond to either
$G_{2b^{\prime\prime}}$ or $G_{3b}$ subclasses, which have also been
classified in an earlier branch (namely nodes 9 and 12) of our decision
diagram.

Thus, in this branch we can assume that
$I_{\mathrm{d}}^{(\infty)}\neq\text{const}$ and hence by the holomorphic
implicit function theorem, there must then exist some holomorphic function
$F(y)$ such that $\ln\mathcal{K}=F(I_{\mathrm{d}}^{(\infty)})$, which
coincides with the signature relation (83) (recall the simplification
$\lambda=1$). Expressed in terms of the $|\mathbf{K}|$ invariant, the relation
becomes

$\ln(|\mathbf{K}|^{2})=2\operatorname{Re}F\left(I_{\mathrm{d}}^{(i\kappa)}\right),$
(167)

where we recognize the harmonic function $\operatorname{Re}F(y)$ as one of the
invariant isometry class parameters discussed in IC.9. The remaining invariant
parameter discussed in IC.9 is isolated by the following relation, where the
function $\mathcal{A}_{\infty,\operatorname{Re}F}(w)$ from (87) is determined
by $\operatorname{Re}F$ (through holomorphic extension and solving an ODE),

$\mathcal{A}_{\infty,\operatorname{Re}F}(I_{\mathrm{d}}^{(\infty)},I_{\mathrm{e}}^{(\infty)})=c>0$
(168)

with the right-hand side constant. Together, these parameters isolate the
isometry class $G_{2b^{\circ};c,\operatorname{Re}F}$.

Together with the data in Figure 2 and Table LABEL:tab:pp-wave-ideal (which
collects the complete IDEAL characterization identities that can be read off
from Figure 2 for each isometry class) this completes the IDEAL
characterization of regular highly symmetric pp-wave geometries. ∎

##  5 Discussion

We have investigated the necessity of traditional scalar polynomial curvature
invariants in the IDEAL approach to the classification of spacetimes, by
considering the class of vacuum pp-wave spacetimes. These spacetimes belong to
the larger class of VSI spacetimes for which all polynomial scalar curvature
invariants vanish.

To hold with the primary principles of the IDEAL approach, we relaxed the
constraint on the construction of scalar invariants by including conditional
invariant tensors and more specifically, conditional invariant scalars as
permitted quantities. A conditional invariant tensor or scalar arises when two
IDEAL tensors are proportional. With this modification, we are able to
characterize all of the highly-symmetric vacuum pp-wave spacetimes in an IDEAL
manner and more finely classify these spacetimes into formfactor classes.

For a given highly symmetric vacuum pp-wave formfactor, there may be
parameters or functions that must be fixed to specify a particular metric.
This presents the problem of sub-classification within a given formfactor
class and whether the IDEAL approach presented here is applicable. We have
attempted to determine a minimal set of IDEAL quantities that will pick out
the parameters and functions in the metric formfactor for each of the possible
symmetry sub-classes. This is possible to do for almost all of the formfactor
classes, except in the case of $G_{3d;\kappa,\gamma}$ where the constant
$\gamma$ only appears in the form of $e^{2i\gamma}$ in every conditional
invariant. Thus, the two cases where $e^{i\gamma}=\pm 1$ cannot be
distinguished using an IDEAL approach. In every other subclass, it is possible
to uniquely characterize a solution in terms of IDEAL quantities.

## Appendix

In this appendix, we review some useful coordinate transformations for the
modest generalizations of those formfactor classes that require a signature
function for the function $A(u)=e^{B(u)}$. In addition to the transformations

$A(u)\left\\{\begin{smallmatrix}\zeta^{k}\\\
\ln\zeta\end{smallmatrix}\right\\}\mapsto\overline{A}(u)\left\\{\begin{smallmatrix}\zeta^{\bar{k}}\\\
\ln\zeta\end{smallmatrix}\right\\},$

we have the following expressions:

$A(u)\mapsto\begin{cases}a^{2}e^{ik\beta}A\left(a(u+u_{0})\right)\zeta^{k}+a^{2}e^{i\beta}A_{1}\left(a(u+u_{0})\right)\zeta&(k\neq
0,1,2)\\\ &~{}z=A_{0}=0,\\\
a^{2}A\left(a(u+u_{0})\right)\ln\zeta+a^{2}e^{i\beta}A_{1}\left(a(u+u_{0})\right)\zeta&(k=0)\\\
&~{}z=A_{0}=0,\\\ a^{2}e^{2i\beta}A\left(a(u+u_{0})\right)\zeta^{2}&(k=2)\\\
&~{}A_{1}=A_{0}=0,\end{cases}$ (169)

where on the right we indicate which integration parameters can be set to zero
by appropriately choosing the gauge parameters $c(u)$, $g(u)$ and $h(u)$ from
(11), with the dependence on the remaining gauge parameters $\beta$, $a$ and
$u_{0}$ explicitly shown. We will need to distinguish two special cases, where
$\operatorname{Re}\dot{B}e^{-\frac{1}{2}\operatorname{Re}B}=\alpha^{-1/2}$ for
$\alpha>0$, in which $\operatorname{Re}B(u)$ no longer depends on the residual
gauge parameter $a$,

$A(u)\mapsto\begin{cases}\frac{\alpha}{(u+u_{0})^{2}}e^{i\operatorname{Im}B\left(a(u+u_{0})\right)}\ln\zeta+a^{2}A_{1}\left(a(u+u_{0})\right)\zeta&(k=0),\\\
\frac{\alpha}{(u+u_{0})^{2}}e^{i\operatorname{Im}B\left(a(u+u_{0})\right)+2i\beta}\zeta^{2}&(k=2),\end{cases}$
(170)

and also the case where
$\operatorname{Re}\dot{B}e^{-\frac{1}{2}\operatorname{Re}B}=0$ and
$\operatorname{Im}\dot{B}e^{-\frac{1}{2}\operatorname{Re}B}=\kappa$ for
$\kappa\geq 0$,

$A(u)\mapsto\begin{cases}e^{i\kappa a(u+u_{0})+2\ln
a}\ln\zeta+a^{2}A_{1}\left(a(u+u_{0})\right)\zeta&(k=0),\\\ e^{i\kappa
a(u+u_{0})+2\ln a+2i\beta}\zeta^{2}&(k=2).\end{cases}$ (171)

We will use the residual gauge freedom to set $A_{1}(u)=A_{0}(u)=0$ and choose
the phase of $\lambda$ so that $\lambda>0$. Hence, the remaining equation for
$\ddot{B}(u)$ integrates to $A(u)=e^{B(u)}=e^{\beta u+\gamma}$, with complex
constants $\beta$, $\gamma$. Under the remaining residual gauge freedom,

$A(u)\mapsto a^{2}e^{2\lambda(bu+c)}A\left(a(u+u_{0})\right),$ (172)

with $a$, $u_{0}$ real and $b$, $c$ complex. The latter two constants appear
because setting $\partial_{u}^{2}h(u)=A_{1}(u)$ determines $h(u)$ uniquely
only up to the addition of $bu+c$ with $b$ and $c$ arbitrary complex
constants. Clearly, the integration constants $\beta$ and $\gamma$ can be
absorbed by the residual gauge freedom.

## References

  * [1] L. Bel, “Sur la radiation gravitationnelle,” Comptes Rendus de l’Académie des Sciences (Paris) 247 (1958) 1094–1096. https://gallica.bnf.fr/ark:/12148/bpt6k7258/f122.item.
  * [2] L. Bel, “Les états de radiation et le problème de l’énergie en relativité générale,” Cahiers de Physique 16 (1962) 59–80. Reprinted in Gen. Rel. Grav. 32 (2000) 2047-2078.
  * [3] S. Bhagwat, M. Okounkova, S. W. Ballmer, D. A. Brown, M. Giesler, M. A. Scheel, and S. A. Teukolsky, “On choosing the start time of binary black hole ringdowns,” Physical Review D 97 (2018) 104065.
  * [4] M. Á. G. Bonilla and J. M. M. Senovilla, “Some properties of the Bel and Bel-Robinson tensors,” General Relativity and Gravitation 29 (1997) 91–116.
  * [5] H. W. Brinkmann, “Einstein spaces which are mapped conformally on each other,” Mathematische Annalen 94 (1925) 119–145.
  * [6] G. Canepa, C. Dappiaggi, and I. Khavkine, “IDEAL characterization of isometry classes of FLRW and inflationary spacetimes,” Classical and Quantum Gravity 35 (2018) 035013, arXiv:1704.05542.
  * [7] A. Coley, D. McNutt, and R. Milson, “Vacuum plane waves: Cartan invariants and physical interpretation,” Classical and Quantum Gravity 29 (2012) 235023, arXiv:1210.0746.
  * [8] A. Coley, S. Hervik, and N. Pelavas, “Spacetimes characterized by their scalar curvature invariants,” Classical and Quantum Gravity 26 (2009) 025013, arXiv:0901.0791.
  * [9] A. Coley, S. Hervik, and N. Pelavas, “Lorentzian manifolds and scalar curvature invariants,” Classical and Quantum Gravity 27 (2010) 102001, arXiv:1003.2373.
  * [10] J. Ehlers and W. Kundt, “Exact solutions of the gravitational field equations,” in Gravitation: An introduction to current research, L. Witten, ed., pp. 49–101.  John Wiley and Sons, New York, 1962.
  * [11] J. J. Ferrando and J. A. Sáez, “On the classification of type D spacetimes,” Journal of Mathematical Physics 45 (2002) 652–667, arXiv:gr-qc/0212086.
  * [12] J. J. Ferrando and J. A. Sáez, “An intrinsic characterization of the Schwarzschild metric,” Classical and Quantum Gravity 15 (1998) 1323–1330.
  * [13] J. J. Ferrando and J. A. Sáez, “An intrinsic characterization of the Kerr metric,” Classical and Quantum Gravity 26 (2009) 075013, arXiv:0812.3310.
  * [14] J. J. Ferrando and J. A. Sáez, “An intrinsic characterization of spherically symmetric spacetimes,” Classical and Quantum Gravity 27 (2010) 205024, arXiv:1005.1780.
  * [15] J. J. Ferrando and J. A. Sáez, “Intrinsic, deductive, explicit, and algorithmic characterization of the Szekeres-Szafron solutions,” Physical Review D 97 (2018) 044026, arXiv:1802.07066.
  * [16] J. J. Ferrando, J. A. Morales, and J. A. Sáez, “Covariant determination of the Weyl tensor geometry,” Classical and Quantum Gravity 18 (2001) 4939.
  * [17] M. B. Fröb, T.-P. Hack, and I. Khavkine, “Approaches to linear local gauge-invariant observables in inflationary cosmologies,” Classical and Quantum Gravity 35 (2018) 115002, arXiv:1801.02632.
  * [18] A. García-Parrado Gómez-Lobo, “Local non-negative initial data scalar characterization of the Kerr solution,” Physical Review D 92 (2015) 124053, arXiv:1510.07561.
  * [19] A. García-Parrado Gómez-Lobo and J. A. Valiente Kroon, “Initial data sets for the Schwarzschild spacetime,” Physical Review D 75 (2007) 024027, arXiv:gr-qc/0609100.
  * [20] A. G.-P. Gómez-Lobo, “Vacuum type D initial data,” Classical and Quantum Gravity 33 (2016) 175005, arXiv:1602.08075.
  * [21] G. T. Horowitz and A. R. Steif, “Strings in strong gravitational fields,” Physical Review D 42 (1990) 1950.
  * [22] A. Karlhede, “A review of the geometrical equivalence of metrics in general relativity,” General Relativity and Gravitation 12 (1980) 693–707.
  * [23] I. Khavkine, “The Calabi complex and Killing sheaf cohomology,” Journal of Geometry and Physics 113 (2017) 131–169, arXiv:1409.7212.
  * [24] I. Khavkine, “Compatibility complexes of overdetermined PDEs of finite type, with applications to the Killing equation,” Classical and Quantum Gravity 36 (2019) 185012, arXiv:1805.03751.
  * [25] I. Khavkine, “IDEAL characterization of higher dimensional spherically symmetric black holes,” Classical and Quantum Gravity (2019) 045001, arXiv:1807.09699.
  * [26] D. S. Krongos and C. G. Torre, “Geometrization conditions for perfect fluids, scalar fields, and electromagnetic fields,” Journal of Mathematical Physics 56 (2015) 072503, arXiv:1503.06311.
  * [27] M. Mars, “On local characterization results in geometry and gravitation,” in From Riemann to Differential Geometry and Relativity, L. Ji, A. Papadopoulos, and S. Yamada, eds., ch. 18, pp. 541–570.  Springer, Berlin, 2017.
  * [28] D. McNutt, Degenerate Kundt Spacetimes and the Equivalence Problem.  PhD thesis, Dalhousie University, 2013.  http://hdl.handle.net/10222/21772.
  * [29] R. Milson, D. McNutt, and A. Coley, “Invariant classification of vacuum pp-waves,” Journal of Mathematical Physics 54 (2013) 022502, arXiv:1209.5081.
  * [30] M. Okounkova, “Revisiting non-linearity in binary black hole mergers,” 2020.  arXiv:2004.00671.
  * [31] P. Olver, Equivalence, Invariants and Symmetry.  Cambridge University Press, Cambridge, UK, 1995.
  * [32] V. Pravda, A. Pravdová, A. Coley, and R. Milson, “All spacetimes with vanishing curvature invariants,” Classical and Quantum Gravity 19 (2002) 6213, arXiv:gr-qc/0209024.
  * [33] H. Stephani, D. Kramer, M. MacCallum, C. Hoenselaers, and E. Herlt, Exact solutions of Einstein’s field equations.  Cambridge University Press, 2009.
  * [34] J. M. Stewart and M. Walker, “Perturbations of Space-Times in general relativity,” Proc. R. Soc. Lond. A. Mathematical and Physical Sciences 341 (1974) 49–74.
  * [35] L. Wylleman, L. F. O. Costa, and J. Natário, “Poynting vector, super-poynting vector, and principal observers in electromagnetism and general relativity,” Classical and Quantum Gravity 38 (2021) 165009, arXiv:2007.15384.