# On the State of the Art in Authorship Attribution and
Authorship Verification

Jacob Tyo1,2, Bhuwan Dhingra3, and Zachary C. Lipton2
1DEVCOM Army Research Laboratory
<EMAIL_ADDRESS>
2Carnegie Mellon University
<EMAIL_ADDRESS>
3Duke University
<EMAIL_ADDRESS>

###### Abstract

Despite decades of research on authorship attribution (AA) and authorship
verification (AV), inconsistent dataset splits/filtering and mismatched
evaluation methods make it difficult to assess the state of the art. In this
paper, we present a survey of the fields, resolve points of confusion,
introduce Valla that standardizes and benchmarks AA/AV datasets and metrics,
provide a large-scale empirical evaluation, and provide apples-to-apples
comparisons between existing methods. We evaluate eight promising methods on
fifteen datasets (including distribution shifted challenge sets) and introduce
a new large-scale dataset based on texts archived by Project Gutenberg.
Surprisingly, we find that a traditional Ngram-based model performs best on 5
(of 7) AA tasks, achieving an average macro-accuracy of $76.50\%$ (compared to
$66.71\%$ for a BERT-based model). However, on the two AA datasets with the
greatest number of words per author, as well as on the AV datasets, BERT-based
models perform best. While AV methods are easily applied to AA, they are
seldom included as baselines in AA papers. We show that through the
application of hard-negative mining, AV methods are competitive alternatives
to AA methods. Valla and all experiment code can be found here:
https://github.com/JacobTyo/Valla

##  1 Introduction

The statistical analysis of variations in literary style between one writer or
genre and another, commonly known as _stylometry_ , dates back as far as 500
AD, when the Hebrew Old Testament was studied and standardized by Tiberias at
Palestine (Binongo, 1996). Computer-assisted stylometry first emerged in the
early 1960s, when Mosteller and Wallace (1963) explored the foundations of
computer-assisted authorship analysis. Today automated tools for authorship
analysis are common, finding practical use in the justice system to analyze
evidence (Koppel et al., 2008), among social media companies to detect
compromised accounts (Barbon et al., 2017), to link online accounts that
belong to the same individual (Sinnott and Wang, 2021), and in a variety of
contexts to detect plagiarism (Stamatatos and Koppel, 2011).

In the modern Natural Language Processing (NLP) literature, two primary
problem formulations dominate the empirical study of methods for determining
the authorship of anonymous or disputed texts: Authorship Attribution (AA) and
Authorship Verification (AV). In AA, the learner is given representative texts
for a canonical set of authors in advance, and expected to attribute a new
previously unseen text of unknown authorship to one of these a priori known
authors. In AV, the learner faces a more general problem: given two texts,
predict whether they were written by the same author or not.

While both problems have received considerable attention (Murauer and Specht,
2021; Altakrori et al., 2021; Kestemont et al., 2021), the state of the art is
difficult to assess owing to inconsistencies in the datasets, splits,
performance metrics, variations in the framing of domain shift across studies,
and the lack of large-scale datasets. For example, a recent survey paper (Neal
et al., 2017) indicates that the state-of-the-art method is based on the
Partial Matching (PPM) text compression scheme and the cross-entropy of each
text with respect to the PPM categories. By contrast, the PAN-2021 competition
(Kestemont et al., 2021) indicates that the state of the art is a hierarchical
bi-directional LSTM with learned-CNN text encodings. Recent work (Fabien et
al., 2020) concludes that the transformer-based language model BERT is the
highest performing AA method. A recent analysis paper (Altakrori et al., 2021)
argue that the traditional approach of character n-grams and masking remains
the best methodology to this day. Each of these sources compares methods
against different baselines, on different datasets (sometimes on just a single
small dataset), and with different problem variations (such as same-topic,
cross-topic, cross-genre, cross-language, etc.).

In this paper, we start by sorting out this fragmented prior work through a
brief survey of the literature. Then, to present a unified evaluation, we
introduce The Valla Benchmark. This benchmark provides standardized versions
of all the common AA and AV datasets with uniform evaluation metrics and
standardized domain shifted test sets. Additionally, we introduce a new large-
scale dataset based on public domain books sourced from Project Gutenberg for
both tasks. Then using this benchmark, we present an extensive evaluation of
eight common AA and AV methods on their respective datasets with and without
domain shift. We also make comparisons _between_ AA and AV methods where
applicable.

Although much recent work indicates that AA and AV are not benefiting from the
recent advancements of pre-trained language models (such as BERT and GPT-2)
commensurate with the gains seen elsewhere in NLP (Kestemont et al., 2021;
Altakrori et al., 2021; Murauer and Specht, 2021; Tyo et al., 2021; Peng et
al., 2021; Futrzynski, 2021), we show that this narrative only appears to
apply to datasets with a limited number of words per class. Furthermore, BERT-
based models achieve new state-of-the-art macro-accuracy on the IMDb62
($98.80\%$) and Blogs50 ($74.95\%$) datasets and set the benchmark on our
newly introduced Gutenberg dataset.

Although the applicability of AV methods to AA problems is frequently
mentioned, few studies place these methods in competition. We provide this
comparison and find that, on AA problems, AA methods outperform AV methods by
over $15\%$ macro-accuracy on average ($76.80\%$ and $59.89\%$ average macro-
accuracy for BERT${}_{\mbox{A}}$ and BERT${}_{\mbox{V}}$ respectively across
five datasets). However, hard negative mining improves the performance of AV
models in the AA setting, increasing the AA performance of BERT${}_{\mbox{V}}$
from $67.21\%$ to $72.42\%$ macro-accuracy on the tested dataset, making it a
competitive alternative.

In summary, we contribute the following:

  * •

A survey of AA and AV.

  * •

An argument for adopting macro-averaged accuracy and AUC as the key metrics
for AA and AV, respectively.

  * •

A benchmark that standardizes AA and AV datasets, and places them in
competition.

  * •

State-of-the-art accuracy on the IMDb62 ($98.81\%$) and Blogs50 ($74.95\%$)
datasets.

  * •

A new large-scale dataset with long average text length.

  * •

An evaluation of eight high-performing AA and AV methods on fifteen datasets,
assessing the state of the art for both problems.

  * •

Evidence of the efficacy of hard-negative mining when applying AV methods to
AA problems.

Dataset |  | Text
---
Type
| Typical
---
Setting
iid | $\times_{t}$ | $\times_{g}$ | $\times_{a}$ | $D$ | $A$ | $W$ | $D/A$ | $W/D$ | imb
CCAT50 | News | AA | $\checkmark$ | — | — | — | 5k | 50 | 2.5M | 100 | 506 | 0
CMCC | Various | AA | $\checkmark$ | $\checkmark$ | $\checkmark$ | — | 756 | 21 | 454k | 36 | 601 | 0
Guardian | Opinion | AA | $\checkmark$ | $\checkmark$ | $\checkmark$ | — | 444 | 13 | 467k | 34 | 1052 | 6.7
IMDb62 | Reviews | AA | $\checkmark$ | — | — | — | 62k | 62 | 21.6M | 1000 | 349 | 2.6
Blogs50 | Blogs | AA | $\checkmark$ | — | — | — | 66k | 50 | 8.1M | 1324 | 122 | 553
BlogsAll | Blogs | AV | $\checkmark$ | — | — | — | 520k | 14k | 121.6M | 37 | 233 | 90
PAN20 & 21 | Various | AV | $\checkmark$ | $\checkmark$ | — | $\checkmark$ | 443k | 278k | 1.7B | 1.6 | 3922 | 2.3
Amazon | Reviews | AV | $\checkmark$ | $\checkmark$ | — | — | 1.46M | 146k | 91.9M | 10 | 63 | 0
Gutenberg | Books | AA | $\checkmark$ | — | — | $\checkmark$ | 29k | 4.5k | 1.9B | 6 | 66350 | 10.5
Table 1:  An overview of datasets used for Authorship Attribution (AA) and
Authorship Verification (AV). iid is an i.i.d. split, $\times_{t}$ is a cross-
topic split, $\times_{g}$ is a cross-genre split, $\times_{a}$ is an unknown
author split, $D$ is the number of documents, $A$ is the number of authors,
$W$ is the number of words, $W/D$ is the average length of documents, $D/A$ is
the average number of documents per author, $W/D$ is the average number of
words per document, and imb is the imbalance of the dataset measured by the
standard deviation of the number of documents per author. $\checkmark$
indicates where the necessary data is available to create a standardized
split, whereas — indicates that the necessary data is not present.

##  2 Brief Survey of the Literature

Neal et al. (2017) provide an overview of AA dataset characteristics and
traditional AA methods. The authors enumerate the wide array of textual
features used for AA (excluding modern methods such as word embeddings and
byte-pair encoding), categorize methods into machine learning, similarity, and
probabilistic models, and provide an evaluation of these techniques on a
single, small dataset. They conclude that the predictions using partial
matching (PPM) method is the state of the art. Bouanani and Kassou (2014)
provide a similar survey focusing on the enumeration of hand-engineered
features used in AA. Stamatatos (2009) discuss traditional AA methods from an
instance-based (comparing one text to another) vs a profile-based (comparing
one text to a representation of all author texts) methodology, including a
computational requirement analysis.

Among notable surveys, Mekala et al. (2018) compare the benefits of the
different traditional textual features; Argamon (2018) detail the problems
with applying many traditional AA methods in real-world scenarios; Alhijawi et
al. (2018) provide a meta-analysis of the field; and Ma et al. (2020) point
out the lack of advances from using transformer-based language models in AA,
calling for more exploration in this area. Critically, all of these prior
surveys exclude recent advances due to deep learning, such as recurrent neural
networks, transformers, word embeddings, and byte-pair encoding. In this brief
survey, we briefly cover more traditional techniques, and then discuss recent
deep-learning-based approaches.

So far, we have outlined the work on AA surveys, but there are none to be
found that focus on AV. The PAN competition overview Kestemont et al. (2021)
is the closest thing that can be viewed as a survey on AV, but it is limited
only to what was seen in the competitions. Notably, each year’s competition
focuses on a single dataset and evaluate methods based on a metric, the choice
of which has varied across the years. Most recently, the winner was chosen
based on a simple average of AUC, F1, F$0.5u$, and C@1.

###  2.1 Datasets

Murauer and Specht (2021) worked towards a benchmark for AA, but overlooked
the applicability of AV methods. Moreover, they do not discuss the domain
shift (systematic statistical differences between train and test sets) present
in many popular datasets. Not only are these datasets not independent and
identically distributed, the test sets often contain novel topics (cross-topic
- $\times_{t}$), genre’s (cross-genre - $\times_{g}$), or authors (unique
authors - $\times_{a}$).

Table 1 shows the wide statistical variability between the different datasets.
Roughly, we can categorize the datasets into small (CCAT50, CMCC, Guardian),
medium (IMDb62, Blogs50, Amazon), and large (BlogsAll, PAN20 & 21, Gutenberg)
datasets. The number of authors, documents, and words in a corpus is
influential, but looking more closely at the number of documents per author
($D/A$) and the number of words per document ($W/D$) gives a better idea of
how hard a corpus is. The larger the number of authors and the less text there
is to work with, the harder the problem. Lastly, we measure the imbalance
($imb$) of datasets based on the standard deviation of the number of documents
per author, most impactful in the AA formulation.

The cross-topic setting is typically used to understand the sensitivity of
models to topical variations. Altakrori et al. (2021) observe that any errors
in this setting can be caused by a failure to capture writing style as well as
topic shift. They then introduce the topic-confusion setting that can more
clearly distinguish the error source. The topic-confusion setting is when all
topics appear in both the training and test sets but the topic of the texts
for each author changes in the test set (i.e. author $A_{1}$ writes on topic
$T_{1}$ and author $A_{2}$ writes on topic $T_{2}$ in the training set, then
in the test set, author $A_{1}$ writes on topic $T_{2}$ and author $A_{2}$
writes on topic $T_{1}$). The authors find that traditional N-gram methods are
the most robust in this setting. While we do not include any experimentation
on topic-confused datasets, Valla does include the topic confusion split they
introduced.

The CCAT50 Lewis et al. (2004), CMCC Goldstein et al. (2008), Guardian
Stamatatos (2013), IMDb62 Seroussi et al. (2014), and PAN20 & PAN21 Kestemont
et al. (2021) are used as they are in prior work, but with the distinction
that we publish our train/validation/test splits to ensure comparability with
future work. The Amazon review corpus He and McAuley (2016) is a very large
dataset, including over 142 million reviews. However, we extract only the
users who have exactly 10 reviews.

The Blogs50 dataset is common and interesting due to its small average text
length. However, the dataset was introduced by Schler et al. (2006) in its raw
form, which correlates to the BlogsAll entry in Table 1. However, the
statistics we present are different than originally published. This
discrepancy is due to a large number of duplicates ($\sim$160,000 exact
duplicates) which we have removed. The most common form of this dataset is
Blogs10 and Blogs50 (the data texts from the “top” 10 and 50 authors
respectively). This is problematic because it isn’t clear how these “top”
authors are selected: the number of documents (Fabien et al., 2020; Patchala
and Bhatnagar, 2018), the number of words, with minimum text length (Koppel et
al., 2011), with spam (or other) filtering Yang and Chow (2014); Halvani et
al. (2017), or as in most cases, not specified (Jafariakinabad and Hua, 2022;
Yang et al., 2018; Zhang et al., 2018; Ruder et al., 2016). In our framework,
we release standard splits and filtering (only removing exact duplicates) for
this dataset.

Machine learning has benefited from larger datasets. However, as highlighted
in Table 1, there is only one common large-scale dataset: PAN20 &
PAN21111These datasets share a training set, differing only in the test set,
so we treat them as a single dataset here. We add a new dataset to this list:
Gutenberg. While some prior work has used Project
Gutenberg222https://www.gutenberg.org as a dataset source (public domain
books), they all use small subsets (Arun et al. (2009) use 10 authors, Brooke
et al. (2015) introduce a generic interface for the data, Gerlach and Font-
Clos (2020) use the 20 most prolific authors, Menon and Choi (2011) use 14
authors, Rhodes (2015) use 6 authors, Khmelev and Tweedie (2001) get a 380
text subset, etc.). Here we have collected all single-author English texts
from Project Gutenberg resulting in almost 2 billion words and a very long
average document length.

###  2.2 Metrics

One of the difficulties in comparing prior work is the use of different
performance metrics. Some examples are accuracy (Altakrori et al., 2021;
Stamatatos, 2018; Jafariakinabad and Hua, 2022; Fabien et al., 2020; Saedi and
Dras, 2021; Zhang et al., 2018; Barlas and Stamatatos, 2020), F1 (Murauer and
Specht, 2021), C@1 Bagnall (2015), recall Lagutina (2021), precision Lagutina
(2021), macro-accuracy Bischoff et al. (2020), AUC Bagnall (2015);
Pratanwanich and Lio (2014), R@8 (Rivera-Soto et al., 2021), and the
unweighted average of F1, F0.5u, C@1, and AUC (Manolache et al., 2021;
Kestemont et al., 2021; Tyo et al., 2021; Futrzynski, 2021; Peng et al., 2021;
Bönninghoff et al., 2021; Boenninghoff et al., 2020; Embarcadero-Ruiz et al.,
2022; Weerasinghe et al., 2021).

[Sentence [Feature Based [Ngram ] [Summary Statistics] [Graph] ] [Embedding
Based [Char Embeddings] [Word Embeddings] [Transformers] ] [Combination] ]

Figure 1: Hierarchy of Prior Work feature extraction methods

The F0.5u, C@1, and Brier Score metrics were popularized by the PAN
competitions to measure the ability of models to abstain from hard samples in
an AV problem formulation. A participant submits a score between 0 and 1 for
each sample with 0 indicating that the texts were written by different authors
and 1 the same author. To abstain from a sample, the participant must submit a
score of exactly $0.5$. However, another one of the metrics used in the
competition is the AUC score, which accounts for how well a model can rank
predictions (i.e. giving a convenient measure of model performance without
concern of a threshold). The PAN scoring ignores any abstained sampled in the
AUC calculation instead of allowing participants to submit two numbers for
each sample, one indicating the model’s score and another indicating if the
model wishes to abstain or not (then the AUC can be measured without the
influence of abstaining while the metrics that account for these non-answers
can still be used as intended). However, we view this level of metric
specificity as more relevant to specific applications and therefore do not
include them in this work.

In both problem formulations, we want to understand the discriminative power
of each model, and we need to be careful to avoid metrics that are determined
too much by performance on a small subset of prolific authors. Thus, we adopt
_macro-averaged accuracy_ for AA (referred to as macro-accuracy), and _AUC_
for AV.

###  2.3 Methods

This section overviews prior methods organized by feature extraction method
(Figure 1).

####  2.3.1 Feature Based

Ngram  The most commonly seen input representation (feature) used in AA and AV
problems are of N-grams. In most cases, the N-grams are counted into a bag-of-
words representation, but in some cases, they are transformed to feature
representations using Convolutional Neural Networks (CNNs) (Shrestha et al.,
2017). Granados et al. (2011) introduced _text distortion_ , which substitutes
out-of-vocabulary items for a “*”. Stamatatos (2018) and Bischoff et al.
(2020) further test these distortion methods and more complex domain-
adversarial methods, showing that the simpler distortion methods are most
effective. Table 2 gives examples of this text distortion.

Given Sentences | The dog’s (and cat’s) house.
---|---
Single Asterisk | The *’s (and cat’s) *.
Multiple Asterisk | The ***’s (and cat’s) *****.
Exterior Characters | The d*g’s (and cat’s) h***e.
Last Two Characters | The *og’s (and cat’s) ***se.
Table 2: Common text distortion methods. The vocabulary in the given example
is {The, and, cats}.

Long considered the best AA/AV method, the Ngram-based _unmasking_ method,
developed by Koppel and Schler (2004), is based on the idea that the style of
texts from the same author differs only in a few features. Given a test text
A, unmasking works by taking all texts from a potential author B and then
building a classifier to predict if the text is from A or B. A small number of
the most meaningful features are removed, and this process is repeated until
there are no features left to remove. At each step, the accuracy is tracked
creating a performance degradation curve. Finally, an SVM is trained to
classify the degradation curve to determine if A and B are from the same
author or not (the ability of the model to distinguish between A and B will
decay quickly if they are from the same author, as only a few features change
between the works of a single author, whereas there are many differences
between the work of different authors. Koppel et al. (2011) later change this
method to keep score of how often each author is predicted after each feature
elimination round, and then make a final prediction based on these scores,
dubbed the imposter’s method, and Bevendorff et al. (2019) use this method for
short texts by oversampling each text.

Seroussi et al. (2011) use Latent Dirichlet Allocation (LDA), comparing the
distance between text representations to determine authorship. They find that
this topic modeling approach can be competitive with the imposter’s method
while requiring less computation. Seroussi et al. (2014) expand on this topic
model approach, and while they presents good results on the PAN’11 dataset,
the performance of the topic modeling approaches lags behind the best methods.

Zhang et al. (2018) introduce a high-performing method that leverages sentence
syntax trees and character n-grams as input to a CNN. Saedi and Dras (2021)
also presents good results with CNN models, but Ordoñez et al. (2020) indicate
that these CNN methods are no longer competitive.

Summary Statistics  While older methods focused on small sets of summary
statistics, more modern methods are able to combine all of these into a single
model. Weerasinghe et al. (2021) provide the best example of this, calculating
a plethora of hand-crafted features and Ngrams for each document (distribution
of word lengths, hapax-legomena, Maas’ a2, Herdan’s Vm, and more). The authors
take the difference between these large feature vectors for two texts and then
train a logistic regression classifier to predict if the texts were written by
the same author or not. Despite its simplicity, this method performs well.

Co-occurance Graphs  Arun et al. (2009) construct a graph that represents a
text based purely on the stopwords (nodes) and the distance between them (edge
weights). Then to compare the two texts, their graphs are compared using the
Kullback-Leibler (KL) divergence. Embarcadero-Ruiz et al. (2022) also
construct a graph for each text i8 but instead represents each node as a
[word, POS_tag] tuple, and each vertex indicates adjacency frequency. After
the graph is created for each text it is encoded into a one-hot representation
and used as input to a LEConv layer. After pooling, the absolute difference
between the two document representations is passed through a five-layer fully
connected network for final scoring.

####  2.3.2 Embedding Based

Char Embedding  Bagnall (2015) use a character-level recurrent neural network
(RNN) for authorship verification by sharing the RNN model across all authors
but training a different head for each author in the dataset. To classify
authors, they calculate the probability that each text was written by each
author, predicting the author with the highest probability. Interestingly, the
authors accidentally ran a version of their system without the RNN (just the
multi-heads) and achieved competitive results.

Ruder et al. (2016) use both CNNs to embed both characters and words for AA.
Their results show that the character-based method outperforms the word-based
approach across several datasets. Compression-based methods, which leverage a
compression algorithm (such as ZIP, RAR, PPM, etc.) to build text
representations which are then compared with a distance metric, fall into this
category as well (Halvani et al., 2017).

Word Embedding  Bönninghoff et al. (2019) leverage the Fasttext pre-trained
word embeddings, concatenated with a learned CNN character embedding, as part
of the input to a hierarchical bi-directional Long Short Term Memory (BiLSTM)
network. Specifically, this combined embedding is used as input to a word-to-
sentence BiLSTM network, whose output is fed into a sentence-to-document
BiLSTM to produce a final document embedding. This neural network structure
runs in parallel for two documents (i.e. as a Siamese network Koch et al.
(2015)), and then optimized according to the modified contrastive loss
function (i.e. the traditional contrastive loss but additionally doesn’t
penalize same-author samples that are sufficiently close in the embedding
space). This method was introduced by Bönninghoff et al. (2019), and then
later modified to include Bayes factor scoring on the output by Boenninghoff
et al. (2020), and by Bönninghoff et al. (2021) to include an uncertainty
adaptation layer for defining non-responses. This was the highest performing
method at the PAN20 and PAN21 competitions (Kestemont et al., 2021).

Jafariakinabad and Hua (2022) attempted to build the equivalent of pre-trained
word embeddings but for sentence structure (i.e. GloVe-like embeddings that
map sentences with a similar structure close together but are agnostic of
their “meaning”). They learn these embeddings by creating a parse-tree for
each sentence using the CoreNLP parser and then passing this parse-tree and a
traditional word-embedded sentence through identical but separate BiLSTMs and
train via contrastive loss. Then, for authorship attribution, they pass the
sentence through both a typical word-embedding LSTM, and also through their
learned sentence structure encoding network, combining their output. The
authors also compare against their prior work (Jafariakinabad and Hua, 2019)
which embeds the POS-tags along with the word embeddings instead of using
their custom structural embedding network. The new work slightly outperforms
their prior method and is more efficient as you can leverage the pretrained
structure encoding network instead of having to always label the POS tags.

CNN’s have also been well explored given word embeddings as input (Hitschler
et al., 2018; Shrestha et al., 2017; Ruder et al., 2016), yet their results
are not among the highest reported.

Transformers  Rivera-Soto et al. (2021) attempt to build universal
representations for AA and AV by exploring the zero-shot transferability of
different methods between three different datasets. The authors train a
Siamese BERT model (Reimers and Gurevych, 2019) on one dataset and then test
the performance on another dataset without updating the model on this new
dataset. Unfortunately, the results seem to indicate more about the underlying
datasets then the ability of these models to uncover a universal authorship
representation.

Manolache et al. (2021) also explore the applicability of BERT to AA by using
BERT embeddings as the feature set for the aforementioned unmasking method.
Comparing this to Siamese BERT, Character BERT (El Boukkouri et al., 2020),
and BERT for classification, they find that simple fine-tuning outperforms the
more complicated unmasking setup.

Following Bagnall (2015), Barlas and Stamatatos (2020) approach the AA problem
by using a shared language model with a different network head for each
author. They then compare different shared language model architectures (RNN,
BERT, GPT2, ULMFiT, and ELMo), finding that pretrained language models improve
the performance of the original RNN architecture. However, the results are all
from the small CMCC corpus.

Tyo et al. (2021) use a Siamese BERT setup with triplet loss and hard-negative
mining for training. Futrzynski (2021) concatenate 28 tokens from each text
and then use BERT’s [CLS] output token for author classification. Peng et al.
(2021) concatenate 256 tokens from each text to produce a 512 token input for
BERT, and then after pooling use linear layers for same/different author
prediction. They repeat this 30 times, sampling different sections of the
input texts, and then average over the 30 predictions for final
classification.

####  2.3.3 Feature and Embedding Based

Fabien et al. (2020) were an early work exploring the applicability of BERT to
authorship attribution. They combine the output of BERT with summary statics
via a logistic regression classifier. The authors find that a BERT-only model
was as effective as a model combining the BERT output with the summary
statistics.

##  3 The Valla Benchmark

In 1440, Lorenzo Valla proved that the Donation of Constantine (where
Constantine I gave the whole of the Western Roman Empire to the Roman Catholic
Church) was a forgery, using word choice and other vernacular stylistic
choices as evidence (Valla, 1922). Inspired by this influential use of AA, we
introduce Valla: A standardized benchmark for authorship attribution and
verification.333Valla can be found here: https://github.com/JacobTyo/Valla
Valla includes all datasets in Table 1, along with others from prior
literature (Klimt and Yang, 2004; Manolache et al., 2022; Overdorf and
Greenstadt, 2016; Altakrori et al., 2021), with standardized splits, cross-
topic/cross-genre/unique author test sets, and usable in either AA or AV
formulation. Valla also includes five method implementations, and we use the
subscript “A” or “V” to distinguish between the attribution and verification
model formulations respectively.

Ngram  Being the best performing method in Altakrori et al. (2021), Murauer
and Specht (2021), Bischoff et al. (2020), and Stamatatos (2018), this method
creates character Ngram, part-of-speech Ngram, and summary statistics for use
as input to an ensemble of logistic regression classifiers. For use in the AV
setting, we follow Weerasinghe et al. (2021) by using the difference between
the Ngram feature vectors of two texts as input to the logistic regression
classifier.

PPM  Originally developed in Teahan and Harper (2003) and best performing in
Neal et al. (2017), this method uses the prediction by partial matching (PPM)
compression model (a variant of PPM is used in the RAR compression software)
to compute a character-based language model for each author (Halvani and
Graner, 2018), and then the cross-entropy between a test text and each author
model is calculated. For use in an AV setup, one text is used to create a
model and then the cross-entropy is calculated on the second text.

BERT  With the highest reported performance on the AA dataset Blogs50 (Fabien
et al., 2020), this method combines a BERT pre-trained language model with a
dense layer for classification. For evaluation, we chunk the evaluation text
into non-overlapping sets of 512 tokens and take the majority vote of the
predictions. For use in the AV setup, the BERT model is used as the base for a
Siamese network and trained with contrastive loss. For evaluation in the AV
setup, we chunk the two texts into $K$ sets of 512 stratified tokens (such
that the first 512 tokens of each text are compared, the second grouping of
512 tokens from each text is compared, etc.), and then take the majority vote
of the $K$ predictions.

pALM  The best performing model in Barlas and Stamatatos (2020) was another
variation on BERT where a different head was learned on top of the BERT
language model for each known author. We refer to this method as the per-
Author Language Model (pALM). To classify a text, it is passed through the
model for each author, and then the author model with the lowest perplexity on
the text is predicted. This is only used in AA formulations as in AV we would
have only a single text to train a network head with.

HLSTM  Originally introduced by Bönninghoff et al. (2019), this method
leverages a hierarchical BiLSTM setup with Fasttext word embeddings along with
custom word embeddings learned using a character level CNN as the base for a
Siamese network. This method was the highest performing at PAN20 and PAN21
Kestemont et al. (2021) and is only used in AV formulations. While this can be
modified to work in AA, we follow prior work and use the original AV setup
only.

All of these methods fall into two categories: the methods that predict an
author class, and the methods that predict text similarity. The methods that
predict an author class (whether via logistic regression, dense layer, SVM,
etc.) need no post-processing. However, the methods that predict similarity
need post-processing both for AA and AV problems. For AA, we build an _author
profile_ by randomly selecting 10 texts from each author and averaging their
embeddings together. Then we can compare the unknown texts to each author
profile and predict the author that is most similar (using the euclidean
distance). For AV, we directly compare the text representations (again using
euclidean distance) and then define a hard threshold based on a grid search on
the evaluation set (although for computing AUC this threshold is irrelevant).

| CCAT50 | CMCC | Guardian | IMDb62 | Blogs50 | PAN20 | Gutenburg AA | Average
---|---|---|---|---|---|---|---|---
Ngram${}_{\mbox{A}}$ | 76.68 | 86.51 | 100 | 98.81 | 72.28 | 43.52 | 57.69 | 76.50
PPM${}_{\mbox{A}}$ | 69.36 | 62.30 | 86.28 | 95.90 | 72.16 | — | — | 55.14
BERT${}_{\mbox{A}}$ | 65.72 | 60.32 | 84.23 | 98.80 | 74.95 | 23.83 | 59.11 | 66.71
pALM${}_{\mbox{A}}$ | 63.36 | 54.76 | 66.67 | — | — | — | — | 26.40
Table 3: Macro-accuracy (%) of the authorship attribution models. The “Average”column represents the average macro-accuracy of each model across all datasets in this table, where — entries are counted as 0%.  |  | CMCC
---
$\times_{t}$
| CMCC
---
$\times_{g}$
| Guard
---
$\times_{t}$
| Guard
---
$\times_{g}$
Ngram${}_{\mbox{A}}$ | 82.54 | 84.13 | 86.92 | 87.22
PPM${}_{\mbox{A}}$ | 52.38 | 57.14 | 69.23 | 72.08
BERT${}_{\mbox{A}}$ | 49.21 | 45.24 | 75.64 | 75.56
pALM${}_{\mbox{A}}$ | 57.14 | 46.03 | 61.79 | 47.22
Table 4: Macro-accuracy (%) of the authorship attribution models. | PAN21 | AmaAV | BlogAV | GutAV
---|---|---|---|---
Ngram${}_{\mbox{V}}$ | 0.9719 | 0.7742 | 0.5410 | 0.8741
PPM${}_{\mbox{V}}$ | 0.7917 | 0.6492 | 0.6230 | 0.8508
BERT${}_{\mbox{V}}$ | 0.9709 | 0.8943 | 0.9201 | 0.9624
HLSTM${}_{\mbox{V}}$ | 0.9693 | 0.8734 | 0.8580 | 0.9147
Table 5: AUC of the AV models on the selected AV datasets. | CCAT50 | CMCC | Guardian | IMDb62 | Blogs50
---|---|---|---|---|---
HLSTM${}_{\mbox{V}}$ | 4.56 | 8.33 | 27.59 | 37.82 | 57.49
(P)HLSTM${}_{\mbox{V}}$ | 13.36 | 16.27 | 38.97 | 59.47 | 11.34
BERT${}_{\mbox{V}}$ | 48.64 | 35.75 | 27.82 | 76.62 | 60.72
(P)BERT${}_{\mbox{V}}$ | 56.80 | 40.87 | 61.41 | 73.17 | 67.21
BERT${}_{\mbox{A}}$ | 65.72 | 60.32 | 84.23 | 98.80 | 74.95
Table 6: Macro-accuracy (%) of the AV models on AA datasets. The (P) indicates that the model was pretrained on the PAN20 training set before fine-tuned on the corresponding dataset.  |  | CMCC
---
$\times_{t}$
| CMCC
---
$\times_{g}$
| Guard
---
$\times_{t}$
| Guard
---
$\times_{g}$
HLSTM${}_{\mbox{V}}$ | 7.94 | 3.18 | 19.23 | 23.33
(P)HLSTM${}_{\mbox{V}}$ | 9.52 | 5.56 | 40.00 | 31.53
BERT${}_{\mbox{V}}$ | 28.85 | 13.49 | 42.31 | 46.53
(P)BERT${}_{\mbox{V}}$ | 33.33 | 19.05 | 43.33 | 54.72
BERT${}_{\mbox{A}}$ | 49.21 | 45.24 | 75.64 | 75.56
Table 7: Macro-accuracy (%) of the authorship verification models on the domain shift AA datasets. The (P) indicates that the model was pretrained on the PAN20 training set before fine-tuned on the corresponding dataset.  | Metric
---
(Formulation)
| AUC
---
(AV)
| Acc
---
(AV)
| Mac-Acc
---
(AA)
BERT${}_{\mbox{V}}$ | 0.9229 | 82.33 | 67.21
BERT${}_{\mbox{V}}$ w/HNM | 0.9276 | 82.72 | 72.42
Table 8: This table compares the performance of the same model
(BERT${}_{\mbox{V}}$), on the same data (Blogs50), just formulated in
different ways, using different performance metrics (column header). w/HNM
represents training with hard negative mining.

##  4 Experiments and Discussion

###  4.1 The State-of-the-Art in Authorship Attribution

After evaluating all methods in Valla on the AA datasets listed in Table 1, we
find that the traditional Ngram method is the highest performing on average as
detailed in Table 3. However, we do see that the BERT${}_{\mbox{A}}$ model
closes the gap on (and can even exceed) the performance of the
Ngram${}_{\mbox{A}}$ method as the size of the training set increases. This
correlation does not hold on the PAN20 dataset, where the best performing
model is still Ngram${}_{\mbox{A}}$. This indicates that the state-of-the-art
AA method is dependent upon the number of words per author available. While we
do not provide a detailed analysis of the data requirements of each method,
our results roughly indicate that Ngram${}_{\mbox{A}}$ is the method of choice
for datasets with less than $50,000$ words per author, while
BERT${}_{\mbox{A}}$ is the state-of-the-art method for datasets with over
$100,000$ words per author.

PPM${}_{\mbox{A}}$ is simple to tune due to few hyperparameters, but it is
both a low performer and it scales poorly to large datasets (rendering it
unusable on the PAN20 and Gutenberg datasets). pALM${}_{\mbox{A}}$ is the
lowest performing method tested, is expensive to train, and scales poorly
resulting in too slow of training to be used on the IMDb62, Blogs50, PAN20,
and Gutenberg datasets.

The macro-accuracy of BERT${}_{\mbox{A}}$ on the IMDb62 and Blogs50 datasets
presents a new state of the art, while defining the initial performance marks
on the GutenbergAA and PAN20 datasets.444These are initial results because the
PAN20 competition was formulated as an AV problem, whereas here we use the AA
formulation The performance on the Blogs50 dataset requires a bit more
analysis due to our filtering of duplicates in the dataset. As a better
comparison to prior reported performance, we first explore the performance of
BERT${}_{\mbox{A}}$ on the Blogs50 dataset _without_ the filtering, and
achieve a macro-accuracy of $64.3\%$. This represents the state-of-the-art
accuracy on a version of the dataset more comparable with prior work (despite
its issues) but indicates the strength of the result reported in Table 3.

Our results on the Guardian and CMCC datasets are hard to compare to prior
work due to the previously mentioned standardization issues, most notably a
i.i.d. split has not been used in prior work. The CCAT50 dataset, on the other
hand, is directly comparable to prior work. Currently, we show best performing
model as the Ngram. However, Jafariakinabad and Hua (2022) reports the
accuracy of a CNN that takes the syntactic tree of a sentence as input as
$83.2\%$ which is better than what we were able to achieve.555CCAT50 is a
balanced dataset, so the macro-accuracy and accuracy are equal. Therefore we
report this previous best result on the performance tracking page of Valla.

###  4.2 The State-of-the-Art in Authorship Attribution under Domain Shift

While the problem of achieving high performance under any domain shift is an
open problem in machine learning, exploration of domain shift in AA and AV
settings is common even if not always explicitly recognized. Table 4 examines
the performance of the same AA models discussed in the previous section but
focuses on the cross-topic and cross-genre test sets of the CMCC and Guardian
datasets. Just as in the i.i.d. setting, the Ngram${}_{\mbox{A}}$ method
dominates in all scenarios. It should be noted that all datasets used in this
domain shift scenario are small, so we cannot verify that the
BERT${}_{\mbox{A}}$ method would begin to dominate as the number of words per
author increases. We leave the exploration of domain shift performance on
larger datasets to future work, although we expect that the
BERT${}_{\mbox{A}}$ model would begin to outperform Ngram${}_{\mbox{A}}$.

###  4.3 The State-of-the-Art in Authorship Verification

In keeping in line with prior work, the distinction between datasets with and
without domain shift is less clear when formulated as an AV problem. The PAN21
test set is comprised of authors that do not appear in the training set.
However the remainder of the datasets (AmazonAV, BlogsAV, and GutenbergAV) are
all iid dataset splits. Table 5 Details the performance of the AV methods on
selected AV datasets.

While we saw the Ngram${}_{\mbox{A}}$ method dominating on most AA datasets,
here we see that the deep learning-based HLSTM${}_{\mbox{V}}$ and
BERT${}_{\mbox{V}}$ methods attain the highest AUC across the board. However,
as we saw the bigger models outperform smaller models when the number of words
per class became large, in AV there are only two classes (same and different
author), and therefore all of the datasets have a very large number of words
per class. Because of this key difference, AV formulations tend to be more
effective for training deep learning methods.

###  4.4 Comparing AA and AV methods

The AA formulation is common in practice, and many AV works are motivated by
the idea that they can be applied to these common AA problems. Yet despite the
prominence of comments indicating how AV is the fundamental problem of AA
(seemingly hinting that applying AV methods to AA problems is easy), there is
no evidence of how well their performance actually transfers. We dive into
this situation in this section. Table 6 shows the performance of
LSTM${}_{\mbox{V}}$ and BERT${}_{\mbox{V}}$ on the i.i.d. AA datasets, both
when trained only on the dataset as well as starting from a pretrained version
of the models (the PAN20 training set was used for pretraining). Here we see
notably lower performance than what was obtained by the AA methods, proving
that the application of AV methods to AA problems is not as straightforward as
it may seem. Table 7 shows this trend continuing in the cross-topic and cross-
genre AA evaluation settings as well.666We note that the lower performance of
the pretrained H-LSTM on Blogs50 than its non-pretrained version is due to the
vocabulary selection. This method chooses its vocabulary based on the
pretraining corpus, and therefore runs the risk of the vocabulary not
transferring well to the intended corpus.

###  4.5 Hard-Negative Mining

As indicated in the previous section, although AV is the fundamental problem
of AA, AV methods do not necessarily perform well under an AA formulation. To
correctly classify a text in the AA setting, a model must make harder
comparisons (i.e., it must compare a text to the hardest negative—the text
that makes the decision the hardest), whereas an AV setting is strictly easier
as it must compare to only a single text. AV problems can be made harder, but
they cannot ever consist of exactly the hardest negatives, because the hardest
negative is model-dependent. This interpretation motivates the exploration of
using hard-negative mining (updating a model during training only on the
hardest examples in each batch) for improving the transferability of AV
methods to AA problems.

The AV formulation is very attractive for training deep models, so bridging
the performance gap would create a strong case for more research emphasis on
AV models. In this section we take a single model (BERT${}_{\mbox{V}}$) and
train two versions of it: one with the contrastive loss and one using triplet
loss with batch hard negative mining (specifically the per-batch hard negative
mining methodology used in Hermans et al. (2017)). Then using our standardized
Blogs50 dataset, we can determine the AV AUC to AA macro-accuracy relation.

Table 8 details these results, showing two key findings. The first is that
high AV AUC does not indicate high AA macro-accuracy, and the second is that
training an AV method with hard negative mining has little effect on its AV
AUC but drastically improves its AA macro-accuracy.

##  5 Conclusion

After a survey of the AA and AV landscapes, we present Valla: an open-source
dataset and metric standardization benchmark, complete with method
implementations. Using Valla and a newly introduced large-scale dataset, we
present an extensive evaluation of AA and AV methods in a wide variety of
common formulations. We achieve a new state-of-the-art macro-accuracy on the
IMDb62 ($98.81\%$) and Blogs50 ($74.95\%$) datasets and provide benchmark
results on the other datasets. Among sufficient words per author ($\sim
100,000$), the state-of-the-art AA method is BERT${}_{\mbox{A}}$, but if ample
data is not available the Ngram${}_{\mbox{A}}$ method proves most effective.
When faced with domain shift, we find the Ngram${}_{\mbox{A}}$ method to be
the most accurate, but notably, all of the domain-shifted datasets are small.

Our results show that the AV problem formulation is more effective for
training deep models, finding the state-of-the-art AV method to be
BERT${}_{\mbox{V}}$. After showing that the high-performing
BERT${}_{\mbox{V}}$ does not perform competitively in AA problems, we explore
the effect of hard-negative mining on its performance and find that with no
degradation in AV performance, it improves the AA macro-accuracy of
BERT${}_{\mbox{V}}$ by over $5\%$, making it a competitive method in the AA
formulation. We hope that Valla encourages more work on understanding the AA
and AV method landscapes, developing new AA/AV methods, and allows for direct
comparison of future findings.

## References

  * Alhijawi et al. (2018) Bushra Alhijawi, Safaa Hriez, and Arafat Awajan. 2018.  Text-based authorship identification-a survey.  In _2018 Fifth International Symposium on Innovation in Information and Communication Technology (ISIICT)_ , pages 1–7. IEEE.
  * Altakrori et al. (2021) Malik Altakrori, Jackie Chi Kit Cheung, and Benjamin C. M. Fung. 2021.  The topic confusion task: A novel evaluation scenario for authorship attribution.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 4242–4256, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Argamon (2018) Shlomo Argamon. 2018.  Computational forensic authorship analysis: Promises and pitfalls.  _Language and Law/Linguagem e Direito_ , 5(2):7–37.
  * Arun et al. (2009) Rajkumar Arun, Venkatasubramaniyan Suresh, and CE Veni Madhavan. 2009.  Stopword graphs and authorship attribution in text corpora.  In _2009 IEEE international conference on semantic computing_ , pages 192–196. IEEE.
  * Bagnall (2015) Douglas Bagnall. 2015.  Author identification using multi-headed recurrent neural networks.  In _Working Notes of CLEF 2015 - Conference and Labs of the Evaluation forum, Toulouse, France, September 8-11, 2015_ , volume 1391 of _CEUR Workshop Proceedings_. CEUR-WS.org.
  * Barbon et al. (2017) Sylvio Barbon, Rodrigo Augusto Igawa, and Bruno Bogaz Zarpelão. 2017.  Authorship verification applied to detection of compromised accounts on online social networks.  _Multimedia Tools and Applications_ , 76(3):3213–3233.
  * Barlas and Stamatatos (2020) Georgios Barlas and Efstathios Stamatatos. 2020.  Cross-domain authorship attribution using pre-trained language models.  In _IFIP International Conference on Artificial Intelligence Applications and Innovations_ , pages 255–266. Springer.
  * Bevendorff et al. (2019) Janek Bevendorff, Benno Stein, Matthias Hagen, and Martin Potthast. 2019.  Generalizing unmasking for short texts.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 654–659, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Binongo (1996) Jose Nilo G Binongo. 1996.  Stylometry.  _Notes and Queries_ , 43(4):448–453.
  * Bischoff et al. (2020) Sebastian Bischoff, Niklas Deckers, Marcel Schliebs, Ben Thies, Matthias Hagen, Efstathios Stamatatos, Benno Stein, and Martin Potthast. 2020.  The importance of suppressing domain style in authorship analysis.  _arXiv preprint arXiv:2005.14714_.
  * Boenninghoff et al. (2020) Benedikt T. Boenninghoff, Julian Rupp, Robert M. Nickel, and Dorothea Kolossa. 2020\.  Deep bayes factor scoring for authorship verification.  In _Working Notes of CLEF 2020 - Conference and Labs of the Evaluation Forum, Thessaloniki, Greece, September 22-25, 2020_ , volume 2696 of _CEUR Workshop Proceedings_. CEUR-WS.org.
  * Bönninghoff et al. (2019) B. Bönninghoff, S. Hessler, D. Kolossa, and R. M. Nickel. 2019.  Explainable authorship verification in social media via attention-based similarity learning.  In _2019 IEEE International Conference on Big Data (Big Data)_ , pages 36–45, Los Alamitos, CA, USA. IEEE Computer Society.
  * Bönninghoff et al. (2021) Benedikt T. Bönninghoff, Robert M. Nickel, and Dorothea Kolossa. 2021.  O2D2: out-of-distribution detector to capture undecidable trials in authorship verification.  In _Proceedings of the Working Notes of CLEF 2021 - Conference and Labs of the Evaluation Forum, Bucharest, Romania, September 21st - to - 24th, 2021_ , volume 2936 of _CEUR Workshop Proceedings_ , pages 1846–1857. CEUR-WS.org.
  * Bouanani and Kassou (2014) Sara El Manar El Bouanani and Ismail Kassou. 2014.  Authorship analysis studies: A survey.  _International Journal of Computer Applications_ , 86(12).
  * Brooke et al. (2015) Julian Brooke, Adam Hammond, and Graeme Hirst. 2015.  Gutentag: an nlp-driven tool for digital humanities research in the project gutenberg corpus.  In _Proceedings of the Fourth Workshop on Computational Linguistics for Literature_ , pages 42–47.
  * El Boukkouri et al. (2020) Hicham El Boukkouri, Olivier Ferret, Thomas Lavergne, Hiroshi Noji, Pierre Zweigenbaum, and Jun’ichi Tsujii. 2020.  CharacterBERT: Reconciling ELMo and BERT for word-level open-vocabulary representations from characters.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 6903–6915, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Embarcadero-Ruiz et al. (2022) Daniel Embarcadero-Ruiz, Helena Gómez-Adorno, Alberto Embarcadero-Ruiz, and Gerardo Sierra. 2022.  Graph-based siamese network for authorship verification.  _Mathematics_ , 10(2).
  * Fabien et al. (2020) Maël Fabien, Esau Villatoro-Tello, Petr Motlicek, and Shantipriya Parida. 2020\.  BertAA : BERT fine-tuning for authorship attribution.  In _Proceedings of the 17th International Conference on Natural Language Processing (ICON)_ , pages 127–137, Indian Institute of Technology Patna, Patna, India. NLP Association of India (NLPAI).
  * Futrzynski (2021) Romain Futrzynski. 2021.  Author classification as pre-training for pairwise authorship verification.  In _Proceedings of the Working Notes of CLEF 2021 - Conference and Labs of the Evaluation Forum, Bucharest, Romania, September 21st - to - 24th, 2021_ , volume 2936 of _CEUR Workshop Proceedings_ , pages 1945–1952. CEUR-WS.org.
  * Gerlach and Font-Clos (2020) Martin Gerlach and Francesc Font-Clos. 2020.  A standardized project gutenberg corpus for statistical analysis of natural language and quantitative linguistics.  _Entropy_ , 22(1):126.
  * Goldstein et al. (2008) Jade Goldstein, Kerri Goodwin, Roberta Sabin, and Ransom Winder. 2008.  Creating and using a correlated corpus to glean communicative commonalities.  In _Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC’08)_.
  * Granados et al. (2011) Ana Granados, Manuel Cebrian, David Camacho, and Francisco de Borja Rodriguez. 2011\.  Reducing the loss of information through annealing text distortion.  _IEEE Transactions on Knowledge and Data Engineering_ , 23(7):1090–1102.
  * Halvani and Graner (2018) Oren Halvani and Lukas Graner. 2018.  Cross-domain authorship attribution based on compression.  _Working Notes of CLEF_.
  * Halvani et al. (2017) Oren Halvani, Christian Winter, and Lukas Graner. 2017.  On the usefulness of compression models for authorship verification.  In _Proceedings of the 12th international conference on availability, reliability and security_ , pages 1–10.
  * He and McAuley (2016) Ruining He and Julian McAuley. 2016.  Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering.  In _proceedings of the 25th international conference on world wide web_ , pages 507–517.
  * Hermans et al. (2017) Alexander Hermans, Lucas Beyer, and Bastian Leibe. 2017.  In defense of the triplet loss for person re-identification.  _arXiv preprint arXiv:1703.07737_.
  * Hitschler et al. (2018) Julian Hitschler, Esther Van Den Berg, and Ines Rehbein. 2018.  Authorship attribution with convolutional neural networks and pos-eliding.  In _Proceedings of the Workshop on Stylistic Variation (EMNLP 2017). September 8, 2017 Copenhagen, Denmark_ , pages 53–28. The Association for Computational Linguistics.
  * Jafariakinabad and Hua (2019) Fereshteh Jafariakinabad and Kien A. Hua. 2019.  Style-aware neural model with application in authorship attribution.  In _2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA)_ , pages 325–328.
  * Jafariakinabad and Hua (2022) Fereshteh Jafariakinabad and Kien A. Hua. 2022.  A self-supervised representation learning of sentence structure for authorship attribution.  _ACM Trans. Knowl. Discov. Data_ , 16(4).
  * Kestemont et al. (2021) Mike Kestemont, Enrique Manjavacas, Ilia Markov, Janek Bevendorff, Matti Wiegmann, Efstathios Stamatatos, Benno Stein, and Martin Potthast. 2021.  Overview of the cross-domain authorship verification task at pan 2021.  In _CLEF (Working Notes)_ , pages 1743–1759.
  * Khmelev and Tweedie (2001) Dmitri V Khmelev and Fiona J Tweedie. 2001.  Using markov chains for identification of writer.  _Literary and linguistic computing_ , 16(3):299–307.
  * Klimt and Yang (2004) Bryan Klimt and Yiming Yang. 2004.  The enron corpus: A new dataset for email classification research.  In _European conference on machine learning_ , pages 217–226. Springer.
  * Koch et al. (2015) Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al. 2015.  Siamese neural networks for one-shot image recognition.  In _ICML deep learning workshop_ , volume 2, page 0. Lille.
  * Koppel and Schler (2004) Moshe Koppel and Jonathan Schler. 2004.  Authorship verification as a one-class classification problem.  In _Proceedings of the twenty-first international conference on Machine learning_ , page 62.
  * Koppel et al. (2011) Moshe Koppel, Jonathan Schler, and Shlomo Argamon. 2011.  Authorship attribution in the wild.  _Language Resources and Evaluation_ , 45(1):83–94.
  * Koppel et al. (2008) Moshe Koppel, Jonathan Schler, and Eran Messeri. 2008.  Authorship attribution in law enforcement scenarios.  _NATO Security Through Science Series D-Information and Communication Security_ , 15:111.
  * Lagutina (2021) Kseniya Vladimirovna Lagutina. 2021.  Comparison of style features for the authorship verification of literary texts.  _Modeling and analysis of information systems_ , 28(3):250–259.
  * Lewis et al. (2004) David D Lewis, Yiming Yang, Tony Russell-Rose, and Fan Li. 2004.  Rcv1: A new benchmark collection for text categorization research.  _Journal of machine learning research_ , 5(Apr):361–397.
  * Ma et al. (2020) Weicheng Ma, Ruibo Liu, Lili Wang, and Soroush Vosoughi. 2020.  Towards improved model design for authorship identification: A survey on writing style understanding.  _arXiv preprint arXiv:2009.14445_.
  * Manolache et al. (2022) Andrei Manolache, Florin Brad, Antonio Barbalau, Radu Tudor Ionescu, and Marius Popescu. 2022.  Veridark: A large-scale benchmark for authorship verification on the dark web.  _arXiv preprint arXiv:2207.03477_.
  * Manolache et al. (2021) Andrei Manolache, Florin Brad, Elena Burceanu, Antonio Barbalau, Radu Ionescu, and Marius Popescu. 2021.  Transferring bert-like transformers’ knowledge for authorship verification.  _arXiv preprint arXiv:2112.05125_.
  * Mekala et al. (2018) Sreenivas Mekala, Vishnu Vardan Bulusu, and Raghunadha Reddy. 2018.  A survey on authorship attribution approaches.  _Int. J. Comput. Eng. Res.(IJCER)_ , 8(8).
  * Menon and Choi (2011) Rohith Menon and Yejin Choi. 2011.  Domain independent authorship attribution without domain adaptation.  In _Proceedings of the International Conference Recent Advances in Natural Language Processing 2011_ , pages 309–315.
  * Mosteller and Wallace (1963) Frederick Mosteller and David L. Wallace. 1963.  Inference in an authorship problem.  _Journal of the American Statistical Association_ , 58(302):275–309.
  * Murauer and Specht (2021) Benjamin Murauer and Günther Specht. 2021.  Developing a benchmark for reducing data bias in authorship attribution.  In _Proceedings of the 2nd Workshop on Evaluation and Comparison of NLP Systems_ , pages 179–188, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Neal et al. (2017) Tempestt Neal, Kalaivani Sundararajan, Aneez Fatima, Yiming Yan, Yingfei Xiang, and Damon Woodard. 2017.  Surveying stylometry techniques and applications.  _ACM Comput. Surv._ , 50(6).
  * Ordoñez et al. (2020) Juanita Ordoñez, Rafael Rivera Soto, and Barry Y Chen. 2020.  Will longformers pan out for authorship verification.  _Working Notes of CLEF_.
  * Overdorf and Greenstadt (2016) Rebekah Overdorf and Rachel Greenstadt. 2016.  Blogs, twitter feeds, and reddit comments: Cross-domain authorship attribution.  _Proc. Priv. Enhancing Technol._ , 2016(3):155–171.
  * Patchala and Bhatnagar (2018) Jagadeesh Patchala and Raj Bhatnagar. 2018.  Authorship attribution by consensus among multiple features.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 2766–2777.
  * Peng et al. (2021) Zeyang Peng, Leilei Kong, Zhijie Zhang, Zhongyuan Han, and Xu Sun. 2021.  Encoding text information by pre-trained model for authorship verification.  In _Proceedings of the Working Notes of CLEF 2021 - Conference and Labs of the Evaluation Forum, Bucharest, Romania, September 21st - to - 24th, 2021_ , volume 2936 of _CEUR Workshop Proceedings_ , pages 2103–2107. CEUR-WS.org.
  * Pratanwanich and Lio (2014) Naruemon Pratanwanich and Pietro Lio. 2014.  Who wrote this? textual modeling with authorship attribution in big data.  In _2014 IEEE International Conference on Data Mining Workshop_ , pages 645–652. IEEE.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Rhodes (2015) Dylan Rhodes. 2015.  Author attribution with cnns.  _Avaiable online: https://www. semanticscholar. org/paper/Author-Attribution-with-Cnn-s-Rhodes/0a904f9d6b47dfc574f681f4d3b41bd840871b6f/pdf (accessed on 22 August 2016)_.
  * Rivera-Soto et al. (2021) Rafael A. Rivera-Soto, Olivia Elizabeth Miano, Juanita Ordonez, Barry Y. Chen, Aleem Khan, Marcus Bishop, and Nicholas Andrews. 2021.  Learning universal authorship representations.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 913–919, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Ruder et al. (2016) Sebastian Ruder, Parsa Ghaffari, and John G Breslin. 2016.  Character-level and multi-channel convolutional neural networks for large-scale authorship attribution.  _arXiv preprint arXiv:1609.06686_.
  * Saedi and Dras (2021) Chakaveh Saedi and Mark Dras. 2021.  Siamese networks for large-scale author identification.  _Computer Speech & Language_, 70:101241.
  * Schler et al. (2006) Jonathan Schler, Moshe Koppel, Shlomo Argamon, and James W Pennebaker. 2006.  Effects of age and gender on blogging.  In _AAAI spring symposium: Computational approaches to analyzing weblogs_ , volume 6, pages 199–205.
  * Seroussi et al. (2011) Yanir Seroussi, Ingrid Zukerman, and Fabian Bohnert. 2011.  Authorship attribution with latent dirichlet allocation.  In _Proceedings of the fifteenth conference on computational natural language learning_ , pages 181–189.
  * Seroussi et al. (2014) Yanir Seroussi, Ingrid Zukerman, and Fabian Bohnert. 2014.  Authorship attribution with topic models.  _Computational Linguistics_ , 40(2):269–310.
  * Shrestha et al. (2017) Prasha Shrestha, Sebastian Sierra, Fabio A González, Manuel Montes-y Gómez, Paolo Rosso, and Thamar Solorio. 2017.  Convolutional neural networks for authorship attribution of short texts.  In _EACL (2)_ , pages 669–674.
  * Sinnott and Wang (2021) Richard Sinnott and Zijian Wang. 2021.  Linking user accounts across social media platforms.  In _2021 IEEE/ACM 8th International Conference on Big Data Computing, Applications and Technologies (BDCAT’21)_ , pages 18–27.
  * Stamatatos (2009) Efstathios Stamatatos. 2009.  A survey of modern authorship attribution methods.  _Journal of the American Society for Information Science and Technology_ , 60(3):538–556.
  * Stamatatos (2013) Efstathios Stamatatos. 2013.  On the robustness of authorship attribution based on character n-gram features.  _Journal of Law and Policy_ , 21(2):421–439.
  * Stamatatos (2018) Efstathios Stamatatos. 2018.  Masking topic-related information to enhance authorship attribution.  _Journal of the Association for Information Science and Technology_ , 69(3):461–473.
  * Stamatatos and Koppel (2011) Efstathios Stamatatos and Moshe Koppel. 2011.  Plagiarism and authorship analysis: introduction to the special issue.  _Language Resources and Evaluation_ , 45(1):1–4.
  * Teahan and Harper (2003) William J. Teahan and David J. Harper. 2003.  _Using Compression-Based Language Models for Text Categorization_. Springer Netherlands, Dordrecht.
  * Tyo et al. (2021) Jacob Tyo, Bhuwan Dhingra, and Zachary Lipton. 2021.  Siamese bert for authorship verification.  In _Proceedings of the Working Notes of CLEF 2021 - Conference and Labs of the Evaluation Forum, Bucharest, Romania, September 21st - to - 24th, 2021_ , volume 2936 of _CEUR Workshop Proceedings_ , pages 2169–2177. CEUR-WS.org.
  * Valla (1922) Lorenzo Valla. 1922.  Discourse on the forgery of the alleged donation of constantine.  _Latin and English. Trans. Christopher B. Coleman. New Haven_.
  * Weerasinghe et al. (2021) Janith Weerasinghe, Rhia Singh, and Rachel Greenstadt. 2021.  Feature vector difference based authorship verification for open-world settings.  In _Proceedings of the Working Notes of CLEF 2021 - Conference and Labs of the Evaluation Forum, Bucharest, Romania, September 21st - to - 24th, 2021_ , volume 2936 of _CEUR Workshop Proceedings_ , pages 2201–2207. CEUR-WS.org.
  * Yang et al. (2018) Min Yang, Xiaojun Chen, Wenting Tu, Ziyu Lu, Jia Zhu, and Qiang Qu. 2018.  A topic drift model for authorship attribution.  _Neurocomputing_ , 273:133–140.
  * Yang and Chow (2014) Min Yang and Kam-Pui Chow. 2014.  Authorship attribution for forensic investigation with thousands of authors.  In _IFIP International Information Security Conference_ , pages 339–350. Springer.
  * Zhang et al. (2018) Richong Zhang, Zhiyuan Hu, Hongyu Guo, and Yongyi Mao. 2018.  Syntax encoding with application in authorship attribution.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2742–2753, Brussels, Belgium. Association for Computational Linguistics.