\begin{table*}
\centering
\begin{tabular}{lrrrr}
\hline
              & \multicolumn{2}{c}{Symbols per Token}                                  & \multicolumn{2}{c}{Words per Token}                                    \\ \cline{2-5} 
Tokenizer     & \multicolumn{1}{l}{Greek BERT} & \multicolumn{1}{l}{Multilingual BERT} & \multicolumn{1}{l}{Greek BERT} & \multicolumn{1}{l}{Multilingual BERT} \\ \cline{2-5} 
Modern Greek  & 4.52                           & 2.55                                  & 0.72                           & 0.41                                  \\
Ancient Greek & 2.98                           & 1.9                      & 0.46                           & 0.31                                  \\ \hline
\end{tabular}
\caption{In comparison with multilingual BERT, Greek BERT tokenizer shows a  higher number of symbols and words per token for both Modern and Ancient Greek} \label{tab:tok}
\end{table*}
\section{Introduction}
\noindent Authorship attribution through some form of statistical inference dates at least
half of a century back, when Mosteller and Wallace used statistics of short, frequent words to estimate authorship of Federalist Papers, disputed by \citet{mosteller1963inference}. The physicist \citet{fucks1968} was the first who systematically developed the methods that relied on statistical patterns like word or sentence length that have been found to be able to distinguish between different authors. While undeniably successful in many cases, they do not require any deeper understanding of the texts in question and therefore have their natural limitations. For a detailed review of various authorship attribution methods developed further, we refer the reader to \cite{stamatatos2009survey}. These methods rely on statistical patterns like word or sentence length that have been found to be able to distinguish between different authors. While undeniably successful in many cases, they do not require any deeper understanding of the texts in question and therefore have their natural limitations.

Transformer artificial neural networks have shown spectacular success in machine translation and answering search queries by internally extracting, after extensive pretraining, abstract patterns, and long-range dependencies in human language samples through encoder hierarchies \cite{vaswani2017attention}. Therefore, it seems natural to apply such schemes to other tasks that traditionally depended on human language understanding. Thus, this paper uses versions of BERT to investigate questions of authorship in Ancient Greek literature. Such models have already been used for authorship attribution, but the
actual number of cases is still limited. \citet{fabien2020bertaa} demonstrate that fine-tuning of a pretrained BERT  language model with an additional dense layer and a softmax activation allows performing authorship classification. \citet{polignano2020contextualized} use BERT for author profiling in social media and conclude that despite encouraging results in terms of reliability, the computational power required for running such a model is too demanding for the task. These results show that transformers could be successfully used for authorship attribution, yet the application of these models to historical texts is still limited. Specifically, \citet{bamman2020latin} develop BERT for Latin and \cite{assael2019restoring} train an LSTM language model of Ancient Greek. There is also a char-BERT implementation\footnote{https://github.com/brennannicholson/ancient-greek-char-bert}, but we do not know of any public full BERT model trained to work with Ancient Greek. We are also unaware of any examples when BERT was used successfully for authorship attribution of historical documents. 

\section{Data}
The data were taken from the digital history projects at the Chair of
Ancient History of the University of Leipzig. The Plutarch texts were transformed into a digital representation under professional supervision. Data can be obtained from the following free repositories Perseus Digital Library\footnote{https://github.com/PerseusDL/canonical-greekLit} and First Thousand Years of Greek\footnote{https://github.com/ThomasK81/TEItoCEX} as part of Open Greek and Latin\footnote{https://opengreekandlatin.org}. The resulting representation was stored in XML format (TEI guidelines) and enriched with metadata. The XML structure and metadata were removed. The strings were transferred into lowercase letters. The diacritics were removed. We did not touch hyphenation, punctuation, or any multilingual remains, nor did we apply any special language-related transformations.
The resulting data set consists of 1 244 documents with 199 809 paragraphs or 14 373 311 words.

\section{Ancient Greek BERT}
The resulting amount of data was too small to train Ancient Greek BERT from scratch. However, data sets of smaller sizes are routinely used for transfer learning and fine-tuning of transformers. Thus, we suggest obtaining BERT for Ancient Greek via transfer learning on a Masked Language Modelling (MLM) task. One could either use Multilingual BERT\footnote{102 languages including Modern Greek, 110M parameters, see https://github.com/google-research/bert/blob/master/multilingual.md} or Greek BERT\footnote{Modern Greek language, 110M parameters, see
https://huggingface.co/nlpaueb/bert-base-greek-uncased-v1} as a starting model for knowledge transfer. The resulting model could then further be fine-tuned for the task of authorship attribution in Ancient Greek.

\subsection{Tokenizers}
The tokenization of words into sub-word tokens is a crucial preprocessing step that can affect the performance of the model. Up to this point, we were using "words" as a linguistic term, whereas we understand tokens as the output of a tokenization algorithm. Thus, there would be one or more tokens that represent every word. Counting the number of tokens used on average to represent a word or, conversely, an average number of words per token gives an estimate of how fit  the tokenization is for the data set. In particular, the average number of words per token varies from 0 to 1, and the closer it is to 1, the more words are represented with one token. Various researchers have shown that corpus-specific tokenization could be beneficial for an NLP task. For example, \citet{sennrich-etal-2016-neural} show that optimal vocabulary is dependent on the frequencies of the words in the target corpus. \citet{lakew2019controlling} and \citet{aji2020neural} partially discuss the tokenization in the setting of cross-language transfer. Though \citet{aji2020neural} demonstrate that there is no clear evidence that one parent is better than another for cross-lingual transfer learning, they also show that token matching and joint vocabulary \cite{nguyen2017transfer} are the best ways to handle the embedding layer during transfer learning. 

Since each model has its own specific tokenizer before pretraining, one wants to measure how well each of them works with Ancient Greek. To do that, one could take two sample data sets: a sample of Modern Greek
Wikipedia (referred to in Table \ref{tab:tok} as “modern”) and a comparable sample of Ancient Greek (“ancient”  in Table \ref{tab:tok}). Each data sample is tokenized with both the Modern Greek BERT tokenizer\footnote{https://huggingface.co/nlpaueb/bert-base-greek-uncased-v} and the Multilingual BERT tokenizer\footnote{https://huggingface.co/bert-base-multilingual-cased}. One can speculate that the model uses shorter tokens to adopt grammatical information and deal with longer, rarely observed words. In contrast, the representations with longer tokens could be useful for semantically intensive problems. These longer, semantically charged tokens may vary significantly on various downstream tasks. Thus, the average length of a token and the average number of words per token, shown in Table \ref{tab:tok}, could be coarsely used as an estimate of the resulting tokenization. One could claim that the higher these values, the more apt the tokenizer is for the task. Indeed, higher average length of a token and number of words per token mean that longer, more semantically charged tokens could be matched for transfer; see \cite{singh2019bert,aji2020neural,samenko2021fine} for a detailed discussion of various tokenization properties.

It is hard to compare the resulting tokenizations that we obtain for the same vocabulary size. Some of the tokens occur in both tokenizations, yet have different frequencies, and some are unique for one of the tokenizations. We want to emphasize that direct token matching would not be relevant for comparing the models. Indeed, a frequent token not matching might significantly influence the downstream performance, while several low-frequency non-matching tokens might not have any noticeable effect on the downstream performance. For a detailed comparison, we publish the resulting vocabularies along with relative frequencies of the tokens obtained\footnote{https://clck.ru/32HWhK}. 

Looking at Table \ref{tab:tok}, one could conclude that the tokenizer of Modern Greek BERT is a preferable solution for Ancient Greek. However, the higher number of symbols or words per token does not automatically guarantee that the overall performance of the model
after fine-tuning would be superior in terms of performance on a downstream task.
{"value": "4.52", "type": "Data Stat.", "dataset": "Modern Greek", "sub-set/group name": "Greek BERT", "attribute name": "Symbols per Token", "dataset features": {"xx": "yy"}} {"value": "2.55", "type": "Data Stat.", "dataset": "Modern Greek", "sub-set/group name": "Multilingual BERT", "attribute name": "Symbols per Token", "dataset features": {"xx": "yy"}} {"value": "0.72", "type": "Data Stat.", "dataset": "Modern Greek", "sub-set/group name": "Greek BERT", "attribute name": "Words per Token", "dataset features": {"xx": "yy"}} {"value": "0.41", "type": "Data Stat.", "dataset": "Modern Greek", "sub-set/group name": "Multilingual BERT", "attribute name": "Words per Token", "dataset features": {"xx": "yy"}} {"value": "2.98", "type": "Data Stat.", "dataset": "Ancient Greek", "sub-set/group name": "Greek BERT", "attribute name": "Symbols per Token", "dataset features": {"xx": "yy"}} {"value": "1.9", "type": "Data Stat.", "dataset": "Ancient Greek", "sub-set/group name": "Multilingual BERT", "attribute name": "Symbols per Token", "dataset features": {"xx": "yy"}} {"value": "0.46", "type": "Data Stat.", "dataset": "Ancient Greek", "sub-set/group name": "Greek BERT", "attribute name": "Words per Token", "dataset features": {"xx": "yy"}} {"value": "0.31", "type": "Data Stat.", "dataset": "Ancient Greek", "sub-set/group name": "Multilingual BERT", "attribute name": "Words per Token", "dataset features": {"xx": "yy"}}